Fair tests of treatments: a quick guide for journalists

Format: 2-16 Don’t confuse “statistical significance” with “importance”
,
Language/s: 3-4 - How certain is the evidence?
,
Short Description:

When deadlines are pressing, how can journalists tell whether to believe claims about the effects of a new treatment or breakthrough?

Key Concepts addressed:

Details

When deadlines are pressing, how can you tell whether to believe the claims about a new medical treatment or breakthrough?

As journalists you are immensely influential on public understanding of new treatments.  So it’s essential that you can tell fact from fiction.

Researchers rarely say that their research was non-significant or biased, and even more rarely do manufacturers that their product doesn’t work.  This in spite of the fact that about half of the time, this is the truth behind their claims.

Can you tell when this is happening?  This TTi webpage for journalists aims to help you.

Some tools to help journalists:

  1. How can I get fact-checks of health news?
  2. Where can I find trustworthy information?
  3. How can I check stories myself?
  4. Some real life examples
  5. The European Health Journalism website contains a wealth of resources for journalists reporting health stories

Fact check

How can I get fact-checks of health news?

If you don’t have the time or skills to do a full fact-check yourself, there is a range of trustworthy sources of evidence-based analysis of current health news stories.

nhs-choices logo
Behind the Headlines
(NHS, UK NHS)
smc-logo
Before the Headlines
(Science Media Centre, UK)
iqwig-logo
Informed Health Online
(Institute for Quality and Efficiency in Health Care, Germany)
national-elf-logo
The National Elf Service
(Minervation Ltd, UK)
health-news-logo
Health News Review
(Informed Medical Decisions Foundation, USA)
cochrane-logo
Cochrane Summaries
(The Cochrane Collaboration)

We recommend these sources because they formally critically appraise the research behind news stories and they are free from conflicts of interest.

You can use them as sources for stories or cross checking stories, and for learning the pitfalls of medical research and how to spot problems.

The aim of critical appraisal is to determine whether:

  1. The researchers did all they could to eliminate bias from their study
  2. The observed results could have occurred if there was no difference between the treatments (in other words, could they have occurred by chance)?
  3. The reported benefit is important in clinical practice.

Critical appraisal should adopt explicit criteria for evaluating these questions.  The best-known criteria are the ones by the Critical Appraisal Skills Programme (www.casp-uk.net).
Reading matter

Where can I find trustworthy information?

Finding reliable, user-friendly sources of information about clinical conditions and treatments can be difficult.  We recommend regular doses of the following:

nhs-choices logo
NHS Choices contains plain language information about tests, treatments and conditions.
PubMed
PubMed Health provides user-friendly summaries of clinical effectiveness research.
iqwig-logo
The TRIP Database aggregates evidence that has already been appraised by someone else.
nhs-evidence logo
NHS Evidence is great for finding current guidelines, and when you can’t find what you need on TRIP.
smc-logo
Science Media Centre experts can provide advice on specific clinical topics.
cochrane-logo
Cochrane Review Groups comprise evidence-savvy experts in specific clinical topics.

Pieces of the puzzle

How can I check stories myself?

Use Ray Moynihan’s tipsheet checklist to assess the importance and reliability of a claim:

  1. What is the size of the potential benefit offered by the therapy, and for what types of patients is it beneficial?
  2. What are the potential harms associated with the therapy?
  3. What are the links between your sources of information about the therapy and those promoting it?
  4. How strong is the evidence to support the claims being made about the therapy, and how does it relate to other available evidence?
  5. What is the natural history of the condition for which the therapy is being offered, and is there potential for what some have called “disease mongering”?
  6. What are the alternatives to the therapy being offered (e.g., no action or watchful waiting, generic drugs, non-drug options, complementary therapies)?
  7. What are the costs of the therapy and are the potential benefits worth the cost?

“Getting answers to these questions can help reporters craft stories that are more likely to serve the needs of their readers, viewers and listeners.  They can also help reporters identify warning signs that the enthusiasm of researchers or the promotional interests of companies and institutions may be getting ahead of the science.” Commonwealth Fund.  Download the tipsheet as a PDF.

Some real life examples

This section documents historical cases that illustrate the need for good reporting of health claims regarding treatments.

MMR

Complications of measles include blindness, deafness, brain damage and death

Complications of measles include blindness, deafness, brain damage and death, image by Julien Harnies, CC BY 2.0

In 1998, Dr Andrew Wakefield claimed that the combined vaccine for Measles, Mumps and Rubella (MMR) might cause autism in young children.

In spite of the vast swathes of evidence contradicting this claim, and the complete absence of evidence to support it, the claim received enormous media attention.

This led to a substantial drop in the uptake of the combined vaccine, a reduction in levels of immunity and subsequent increases in the incidence of these harmful diseases.

This illustrates:

  • Conflict of interest:  Dr Wakefield had commercial interest in selling single vaccinations against each disease
  • Uncritical acceptance of medical claims by the media, leading to the creation of controversy where none existed
  • Measurable harm from inaccurate reporting:  The incidence of measles has increased as our population’s immunity decreased.  The outbreak of measles in South Wales was one notable example.

Read more:

Other examples

Please see Why do we need fair tests of treatments? which contains numerous examples of the myriad ways in which patients have been harmed by our failure to tests treatments properly.

Print Friendly, PDF & Email

Leave a Reply

Your email address will not be published. Required fields are marked *