archives

Should you trust the latest health news? Here’s how to tell.

May 13, 2014, 6:20 PM UTC

Getty Images

The world is awash in health reporting, some of it good, much of it bad. Even bona fide news organizations are producing stories in newspapers, online, and on TV that are surprisingly subpar. Many overhype medical treatments and procedures, while playing down risks or glossing over poor evidence.

So how can we separate good health reporting from bad? Earlier, I wrote on how to tell if a science news story is hogwash, and much of that applies here. But medical stories can have additional problems because people use them to help make decisions about their own health care.

For years, the independent media watchdog project HealthNewsReview.org has been having experts rate health news stories on 10 criteria, which I’ve paraphrased below. Granted, not every good story will get an absolutely perfect score, but these are great questions to keep in mind:

What’s the total cost of the treatment or procedure?
How often does it benefit people?
How often does it cause harm?
How strong is the evidence that it works (or doesn’t work)?
Has the prevalence of a medical condition been exaggerated (aka disease mongering)?
Are there alternative treatment options, and have they been included in the story?
Is this really a new approach?
Is it available for people right now or not?
Who’s promoting this? Who is the source of the information?
Do the sources have a conflict of interest (such as a doctor receiving fees from a drug company for researching or using its drug)?

In May, journalist Gary Schwitzer, summarized the site’s reviews of 1,889 news stories over seven years in the journal JAMA Internal Medicine. Most of the ones they graded did poorly on properly addressing costs, benefits, harms, the quality of evidence, and alternative options.

The piece also pointed out some other things to watch out for. Here they are, paraphrased, with my explanations below:

1) Risk reduction should be stated in absolute, not relative terms: A 50 percent reduction in deaths has a completely different meaning if the change was from 2 to 1 people out of 100 than if it was from 80 to 40 people out of 100. The absolute risk within a population definitely matters.

2) Observational studies have limitations: This is the perennial correlation vs. causation problem. If a study merely observes a correlation, it can’t tell you what’s causing what. (Check out this story on the relationship between margarine consumption and divorce rates, and other nonsense correlations.) Determining a cause requires an experiment.

3) Anecdotes about individual patients can be deceiving: One lovely anecdote does not a trend make.

4) Stories should address the tradeoffs between the benefits and harms of screening for diseases: Screening more generally will diagnose more people. But that increased testing can also increase risks associated with both the test itself and further medical interventions for people who falsely tested positive (for example, biopsies for breast cancer). Another thing to keep in mind: did catching the problem earlier actually lead to better outcomes? (You can read more about how mammograms aren't as effective as you think in our previous story here.)

5) Note what endpoint was measured and how meaningful it is: Sometimes researchers look at markers (such as the level of a certain protein) rather than what really matters at the end of the day — actual patient health. One good example is cholesterol levels vs. heart attacks and strokes. Markers can be a handy tool, but it’s not the full story.

6) Watch out for stories recycled from press releases or with only one source: There’s a lot of recycling going on these days. Curious if your news outlet is one of the guilty? You can generally Google your way to an original press release and find out.

7) Watch out for enthusiastic coverage of new medical technology: It’s new and shiny and costs a zillion dollars! But does it actually make people healthier than other options would?

8) Watch out for overly flattering business stories: Just because investors believe in a company doesn’t mean that its health products or services have proven themselves. What does the data actually say?

Further reading:

The full article should be available for free until November 2014 here.
HealthNewsReview.org specifically reviews news stories that make safety or effectiveness claims about health care interventions. It's doing fewer reviews than it used to because of a loss of funding, but you can find occasional new ones on its blog. The site also has a thorough explanation for each of its ten ratings criteria.