Digital media content creators have an ethical responsibility to journalistic integrity.
All of them do; all of us… including myself.


At the last few international conferences I attended (SqlBits and Microsoft Fabric Conference), I had several interesting discussions with various content creators about their experience creating content about software products and services. I also talked with many attendees who consume the content and who rely on this content for self-education, to stay up-to-date, and even to make important company decisions – decisions they sometimes regret because of incorrect or incomplete information they absorbed. These discussions have brought forth an important topic I reflect upon in this blog post.

With digital and social media, the market for IT product reviews has undergone a significant shift. In the past, the primary source of such information was a collection of specialized IT magazines focusing on hardware and software. These publications targeted specific audiences and reviewed tools for specific markets, such as SQL Server for database administrators and Visual Studio for developers.

The elimination of intermediaries in digital marketing has also impacted IT magazines, previously the middleman between producers and consumers of IT products and services. Most IT magazines have disappeared, and the few that remain focus only on large, general markets. However, there is still a lot of content available: it is just produced by many content creators who are rarely structured as a company. Many companies produce marketing and educational content targeting potential customers directly, but they make a fraction of the content generated by independent content creators. Blogs, LinkedIn, Medium, Reddit, YouTube, TikTok, and many other social media platforms mainly deliver content produced by “users” of tools (like developers, database administrators, business analysts, …) who share their opinions with readers and subscribers.

This change is a process that took years and is not complete; it is a continuous, incremental evolution. However, only some content creators realize or understand the importance of this change. They also may need to acknowledge their responsibility as content creators.

As a content creator, I find myself at the heart of the topic of this blog post. I am first to admit that I should be more mindful of the impact of the articles and videos I publish. I extend this invitation to the broader community of content creators, as I believe many of us may not fully grasp the scale and importance of these concepts.

With a bigger audience comes greater responsibility

We all started with zero followers, zero subscribers, zero listeners, and zero readers. Whatever the content you offer at that point, there are no side effects. But the video you record, the blog you write, and the podcast you stream may very likely stay there, always accessible, and also accessible to your future audience when there is one.

Search engines will also include your content in future searches. Your small sentence or thumbnail could perfectly match a future complex search, and users desperately looking for particular information will land on your content. You get new followers, followers who trust you because you helped them, even though you do not know that.

This is a story that everybody knows. Details change, but the screenplay follows the same script. Over time, your content might even become part of the decision-making process in a small or large organization. It could increase a company’s revenues or decrease another company’s sales. It’s the way it works, but in this game, every content creator influences the market much more than his/her actual efforts and direct feedback would suggest.

Thus, we should all pay attention to what we do. Words (and videos) have consequences.

Reviews

Should we produce reviews? Yes! Why not?

There is nothing wrong with writing a review of a product. However, the consumer of the content should be aware of the conditions that contributed to producing the review.

For example, I have probably been using ZoomIt since its first version, and I constantly use it during demos in classrooms and videos. ZoomIt is a (free) product that I can endorse because I am a happy user of it. These are the reviews we expect to read as consumers: someone who knows a product very well can be trusted to accurately describe its pros and cons.

However, we only have a deep experience on some of the products. But sometimes we feel the need to share our opinion of a product we do not know well, but we tried to use. Or a product that has just been released, so we want to write something about it even though we have yet to really test it. Makes sense. However, we should warn the audience about our limited experience. We should make it clear that we have superficial knowledge; the trustworthiness of a review written based on years of experience cannot be the same as a first impression of a new technology.

We should also share the why. Why are we writing a review? Is it because we have used a product for so many years and want to share our deep expertise? Or is it because we are excited about a new tool or feature? Or because someone else asked us to do so?

We should inform the audience. There is nothing wrong with reviewing a new product. But there is a difference between trying a product because we are curious about a new tool and trying a product because the vendor asked us (or paid us) to do so. The review might be legitimate, but it is unfair to rely on our audience’s trust in a review that has been requested from us without informing this audience. It is also important to make a clear distinction between a spontaneous review and a requested (or paid) endorsement of a product, person, or service.

Conflict of interest

This is an extension of the solicited review case (described before) but can also be applied to benchmarks and comparisons (described later). You must disclose when you have a conflict of interest. Examples of conflict of interest include:

  • You were paid to produce the content, either directly (in money) or indirectly (with free software, perks, or career opportunities).
  • You have a professional affiliation with the software, such that they are your employer or pay for your services (as a vendor).
  • You stand to benefit—directly or indirectly—by endorsing the software. For instance, if you own a commercial stake in the company producing or selling the software.

In small communities, conflicts of interest are very likely. It is unavoidable. However, the IT audience is made up of smart people, so disclosing any conflict of interest is the way to go. It also helps your content become more trustworthy, as you are honest and authentic with your audience.

I am personally guilty of not doing enough. I am a cofounder of Tabular Tools, the company that produces DAX Optimizer, and I assume that everybody knows that. After all, I have mentioned these facts several times over the last few months. However, I understand that a random reader of my blog might need to realize that if I speak so highly of DAX Optimizer, that is also because I am one of the developers who contributed to its creation! I honestly think it is a good product, but now you know I could be biased, so… you should try it to ensure you understand the facts!

However, I will review existing articles and videos to disclose the existing conflict of interest properly.

Benchmarks

Producing content containing benchmarks involves many risks. The software’s license agreement often explicitly forbids publishing specific benchmarks (like point 8 in SQL Server licensing). I do not remember the last time this condition was enforced by a vendor, but I am sure it has happened in the past and could happen again. Thus, to avoid experiencing unwanted consequences of your benchmarking, read the license agreement conditions carefully before you publish these results.

When benchmarks and performance comparisons between products are allowed, you still must consider the possible misinterpretation of your content. Is the benchmark relevant for a generic condition, or is it something very specific to the model you used? Even just suggesting a best practice for a product can lead many users to non-ideal choices that might negatively impact them, rather than yield an improvement.

When you produce a benchmark, you can help the audience interpret it by doing things like:

  • Including multiple technical replicates to avoid your results being due to random chance. When you share the results, show the aggregate and share how many tests you did.
  • Including multiple test replicates with different conditions. This ensures that your results are applicable across various scenarios, queries, models, and data. If your results are specific to your test conditions, or you haven’t had the time or interest to test more broadly, state that, as it affects how interpretable your results are to a reader’s scenario.
  • Stating the impact and relevance of these findings. An increase of 1% in query time might be measured, but is it significant? Realistically, will a user ever notice that difference? Probably not.

Enable the audience to reproduce your benchmark. This is the best way to ensure that the final decision is made on a case-by-case basis. Everybody understands that different conditions could provide different performance results. Providing guidance to replicate the benchmark is the best way to help other people make the right choice.

For instance, you can make your benchmarks more reproducible by doing things like:

  • Providing the test results, source files, and any metadata (like code) you used for the benchmark;
  • Describing the test conditions (such as software configuration and the steps to execute the test) so that someone can replicate them precisely for their scenario;
  • Providing a description of your machine if the test was run locally, or of the server if it wasn’t.

Benchmarks provide a source for valuable (and enjoyable!) content. However, they can be misinterpreted and used to motivate decisions outside your intended scope.

Conclusions

As a community, we should ensure that the content we produce can be trusted. Describe the reasons why you wrote a product review, ensure that the audience can reproduce a benchmark, and disclose any conflict of interest.

If you see any similarities with the responsibilities of journalists, that’s just it: when you choose to produce content for a public audience, you are a journalist in this world.