User Comments: Do They Affect The Credibility Of Science Stories?

By Tony Van Witsen

Title: Contested Certainty and Credibility: The Effect of Personal Stories and Scientific Evidence in User Comments on News Story Evaluation and Relevance

Author(s) and Year: Amanda Hinnant, Sisi Hu, Yoorim Hong, Rachel Young; 2023

Journal: Science Communication

TL;DR: Online reader comments about news stories have been praised for democratizing public access and criticized for undermining journalistic authority. This research sought to measure the impact of reader comments. Its findings suggest that they may negatively affect the credibility of science reporting.

Why I chose this paper: As a journalist, I frequently wondered what spurred people to weigh in with their two cents’ worth. Many comments I read didn’t seem to fit with any standard ideological preconceptions. This research intrigued me because it tried to grasp what’s going on in reader-land and what it means for science reporting.

The problem

Twenty-four years ago, in 1999, the Fresno Bee started a revolution in journalism by placing a “forum box” at the end of online news stories where readers could add their comments and read the comments of others. Within days, web traffic increased by 30 percent and stayed there. As more media added these comment sections and pushed their reporters to manage them, some praised this innovation as a triumph of openness and democracy, ending the era in which journalists spoke into the void with no response.

That was the hope, anyway. Researchers who studied comments have found incivility, bigotry and hate in comments sections, confirming the fears of some journalists. And their participatory culture displaced knowledge authority away from journalists who presumably had evidence to support what they wrote. By 2022, many news organizations had dropped the comment forum from their news pages, fencing them off onto related social media platforms.

Research design and questions

Amanda Hinnant, Sisi Hu, Yoorim Hong, and Rachel Young wondered if reader comments were a failed experiment in bad influence. They were especially concerned about the context of science news, where both journalists and researchers have noted the tendency of user comments to undermine the authority of science. From multiple past studies on vaccination, the authors knew comments that were positive about flu vaccines led to lower perception of vaccine risk, more positive attitudes toward vaccination, and greater likelihood of being vaccinated, but negative comments about vaccines lowered a story’s perceived credibility. To further probe these findings, they conducted an experiment on whether user comments, separate from the story, could shape audience reactions.

Research design and questions

They asked 426 American adults (29.3% male, 70.4% female) to read a story (Figure 1) about how overprescription of antibiotics can lead to antibiotic resistance. Some read the story with no comments at all.

Figure 1: The story

Others read the story with four categories of comments: (See examples, Figure 2)

  • Supporting the story with anecdotal evidence from the commenters’ own experience;
  • Supporting the story with scientific evidence;
  • Dissenting from the story with anecdotal evidence; and
  • Dissenting from the story with scientific evidence.

Figure 2

The authors then asked subjects to say whether they found the story:

  • Personally relevant;
  • Accurate and credible; and
  • Likable;

and whether they were:

  • Likely to share it;
  • Certain of the story’s conclusions; and
  • Worried about antibiotic-resistant infections, personally.

Findings

The story with anecdotal comments was rated significantly more credible than the one with scientific comments, as well as more likable. However, the kind of comments did not make a story more relevant, less uncertain, or more likely to be shared. Nor did the comments make antibiotic-resistant infections seem riskier.

The antibiotic resistance story with supportive comments was considered more relevant, more credible, and likable than the one with dissenting comments. People were also more willing to share the story with supportive comments and worried about vaccine risk more. They did not, however, see it as more or less uncertain than the story with dissenting comments.

The authors concluded that reader comments affect how people view science stories, but in different ways. Dissenting comments challenging a story’s scientific conclusions had the strongest effect when they cited actual scientific counterevidence. When that happened, people were willing to change their minds about a story’s credibility and relevance.

Purely anecdotal comments, on the other hand, had no more effect on how people felt about vaccines than a story with no comments at all. At the same time, anecdotal comments and supportive comments did make stories seem more credible even if they didn’t change minds. Stories with no comments were sometimes seen as being as relevant and certain as stories with only anecdotal comments, which the authors considered a good reason to drop comments sections from new websites. Because supportive comments are nowhere near as frequent as critical comments, the abundance of criticism might reduce the credibility of science reporting. The authors don’t think that’s good for either journalism or science.

Impact

Comments sections may be less frequent on news sites now, but are still everywhere on social media. Though audience feedback was once seen as valuable, there’s growing evidence of how it can hurt journalism’s credibility, especially when it comes to science. Journalists who use social media should be aware that comments, particularly those with dissenting scientific evidence, can reduce their credibility, even when the story isn’t especially controversial. In fact, user comments alone may make stories seem controversial, even when scientists themselves don’t disagree.

This is a problem for science journalism in a way it is not for politics, where it’s taken for granted that ordinary citizens are free to agree or disagree with specific policies and policymakers. In science, where not everyone’s opinion counts equally, citizen dissent, what the authors refer to as people “doing their own research,” can risk the credibility even of matters on which most scientists agree. Journalists should understand how reader comments can affect debate on controversial medical or scientific issues.

Edited by: Héctor Torres Vera; Kay McCallum

Cover image credits: Cottonbro studios (Pexels); Hassan Albari (Pexels); Mali Maeder (Pexels); Steven Soblick