Saturday, 10 February 2018

The 'JAMES H RANDI' Framework: Assessing Science Reporting.

It's vital to acknowledge that most people do not get information about the sciences from peer-reviewed papers, instead relying on the media to disseminate information to them. There's nothing intrinsically wrong with this, but most journalists are not qualified in the sciences, this includes actual science correspondents and content writers for science periodicals like New Scientist and Scientific American. Headlines are often vastly exaggerated or outright false. The articles in question may not reflect the findings of the paper or study they concern because the journalist failed to understand what they were reading, they took the information in their article from a secondary source that was incorrect or they've blatantly misrepresented the findings of the research to fit some other narrative or belief. Of course, there's another possible reason for the article being wrong, it may accurately represent the study it reports, but that study may itself be deeply flawed. So it's vital to review how to assess a news article that relates scientific findings.

https://wattsupwiththat.com/2017/12/30/editorial-narratives-in-science-journalism/n
In what follows I'm going to draw heavily on the work of Kevin McConway and David Spiegelhalter (1), two statisticians, who after getting tired of hearing bogus medical claims on the morning radio, developed a framework to assess the reporting of medical studies in the press. At points, I'm going to generalise to make the points apply beyond the medical sciences. My adapted framework contains eleven questions divided into two categories, study quality and the standard of reporting.

Scoring your article

All the questions in the framework can all be answered 'yes' or 'no' but you'll notice that they are sometimes worded in a quite ungainly fashion. This is because one point is awarded for a 'yes' and zero points are being awarded for a 'no'. Thus the higher an article's score the less trustworthy it is. An article with a score of seven or above should be considered deeply flawed, an article with 10 or more, utter bunkum.


Quality of study.




1. Just Observational? 


Have the researchers made any attempt to control for other variables or have they simply observed a process without interference? Whilst a lack of experimenter tampering may sound like a good thing, failing to apply proper controls make it extremely difficult to link a cause to an effect. Imagine testing a medical intervention but failing to control for other treatments. How can we tell which medical intervention caused an observed improvement?

2. Another Single study? 


 What journalists often fail to realise is that scientific consensus cannot be built upon the outcome of one study. We should establish if the study in question has been successfully replicated, or if the results found reflect those found in other, similar investigations into the same phenomena. 

3. Might there be another explanation for the observed effect?


Is there a confounder that might explain the found results? The experimental controls should allow the researchers to eliminate plausible alternative explanations for an observed effect. Imagine experimenters are testing a new cold remedy. They select two groups, men and women. They give the women the new drug, but not the men. They find that the women in the group administered the medicine tend to recover more quickly than the men and report less extreme symptoms. They conclude the remedy is successful, but they have failed to control for gender. The experiment is confounded.

We should also consider any systematic bias, have the researchers introduced an element into the study that will skew the results in favour of one particular outcome? A striking example of this would be a recent survey issued by the Trump administration comparing voters opinions of the first year of Trump's first term to the first term of Obama's first term (below). You'll immediately notice that the first question has an element missing. Subjects are unable to rate Trump poorly whereas the option is available in the second question which asks subjects to rate Obama's first term(2). This quite laughable omission means that a side by side analysis is unsuitable.


4. Extrapolating Small sample sizes?

We should be extremely wary of studies with small sample sizes, especially those with subjects numbering in the tens rather than the hundreds or even thousands. There are mathematical ways to calculate appropriate sample sizes, but often it's easy enough to do this intuitively. You can't draw conclusions about millions of people based on a study of tens.  For example, consider Andrew Wakefield's withdrawn Lancet study which attempted to establish a link between the MMR vaccine and autism. Wakefield's study group contained five children, clearly not enough to draw conclusions about millions who had received the MMR vaccine. We need to be even more concerned when the conclusions of a series of tests are extrapolated to a much larger population

In relation to sample sizes, it's important to be wary of larger of studies of rare events. For example, a study of a rare illness may involve following millions of people but only an extremely small number of that sample develop the illness in question.

5. Samples not varied enough? 

Related to the previous point, it's not suitable to draw conclusions about a large population based on a sample that isn't varied enough. A good example of this is the study I looked at with the Spooktator crew early last year. The study proposed to show that individuals with strong religious or supernatural beliefs have poor cognitive abilities. The problem was, not only were the sample sizes extremely small but the vast majority of those studied were aged under 25 and female. It's not possible to draw conclusions about millions of believers of all ages and both sexes from such small, unvaried sample sizes. 

Standard of reporting.

6. Half (or less) of the story?

Are the reporters telling you everything? If they are reporting on the harmful effects of a medicine are they pointing out the benefits as well, or vice versa? Are they highlighting a small part of the research and ignoring the bigger picture? Researchers will normally point out flaws with their studies and suggest avenues for further research. Are these elements being covered in the report?

7. Representing risk in a misleading way? 

Watch out for the phrase "higher risk" in a report. If you are told that exposure to a substance doubles your risk of a certain ailment or illness it sounds quite bad. But what if your risk was incredibly low, to begin with? Unfortunately "X doubles the risk of Y" makes a fantastic attention-grabbing headline. This can also be true when considering a stated effect. If some variable makes the chance of a positive outcome more likely, we need to know how likely that outcome was in the first place to know if that is significant or not. To combat this we should be looking at absolute numbers as a sign of good science reporting.

8. An Exaggerated headline? 

Headlines for articles can be difficult to construct, this sometimes means important details are omitted, worse still they can be abandoned in favour of hyperbole. Does the headline of the article actually reflect what is said in the actual report, or is it misrepresentative or manipulated?

A great example of this would be a study published by the International Agency for Research on Cancer (IARC) reflecting the decision to list the radiation from mobile phones as “possibly carcinogenic to humans” or in specific terms to place classify it as a group 2B carcinogen (3). The 2B category is used when there is no specific evidence of a substance or material posing an actual risk, but there have been correlations made in the past. Some other 2B carcinogens include; fuels, laundry detergents and aloe vera.

The Daily Express clearly weren't interested in these details when they reported the IARC's report with the headline: "Shock Warning: Mobile phones can give you cancer" (4).

This headline complete strips the subtlety of the IARC report in favour of hyperbole and blind panic.

9. No Independent Comment?

When considering a scientific study it's vital to remember our first point, single studies do not make the scientific consensus. That means that we should be looking for independent comment from someone in the field of research not involved with the research in question to put our study in context. If an article omits this, it's likely based on promotional material issued by the institution that produced the research, one that has vested interest and may well not be as even-handed as one could hope. This doesn't mean these comments have to be negative, but they should be present.

This leads us to...

10. Does the report rely on public relations puff pieces, or are there considerable personal interests involved?

Are there elements of the report that imply the study is just PR? Who sponsored the research? What was the ultimate aim of the study? Does it fit into a wider scientific context? The answers to these questions are likely to tell you whether you should take the report with a pinch of salt or a shovel. This isn't to say that research that has been paid for by a company or corporation should be immediately disregarded, but it should be viewed with some skepticism. Likewise, research conducted by individuals with considerable personal interests in the research should be considered with suspicion.

For example, Martin Pall's research (5) on the dangers of electromagnetism should be weighted alongside the fact that he sells a range of supplements that he claims to strengthen the biological systems which his research claims EMF 'attacks'(6). Is this conflict of interest mentioned in the report, or in the original paper even?



.

11. Is the original research unavailable?


A great deal of the time, you'll find that the article you're reading doesn't even link to the original study. This means you're going to have to do the legwork yourself. The original study should be searchable by title if this gets you no results try searching by selected keywords. When doing your search, you may be far more likely to find success searching using an academic search programme such as Google Scholar (7). When you find your paper, it may well be hidden behind a paywall. Don't despair, even if this is the case, the abstract will be available for free. You will more likely than not find that this alone is sufficient to find errors in sloppy articles, especially if the author didn't even bother to read the abstract as often is the case!

Conclusion 

If you're slightly worried that all that may be difficult to remember, fear not, I've formed it into a handy mnemonic, JAMES H RANDI after my skeptical hero. You could always rework the framework to spell out the name of your own hero of science or skepticism. I've also formed the questions into a rudimentary scorecard which you can see below and download by following the link in the sources (8). Hopefully, it should make assessing science articles much easier.



Sources

Posted by at
Labels: , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)