The 'JAMES H RANDI' Framework: Assessing Science Reporting.

It's vital to acknowledge that most people do not get information about the sciences from peer-reviewed papers, instead relying on the media to disseminate information to them. There's nothing intrinsically wrong with this, but most journalists are not qualified in the sciences, this includes actual science correspondents and content writers for science periodicals like New Scientist and Scientific American. Headlines are often vastly exaggerated or outright false. The articles in question may not reflect the findings of the paper or study they concern because the journalist failed to understand what they were reading, they took the information in their article from a secondary source that was incorrect or they've blatantly misrepresented the findings of the research to fit some other narrative or belief. Of course, there's another possible reason for the article being wrong, it may accurately represent the study it reports, but that study may itself be deeply flawed. So it's vital to review how to assess a news article that relates scientific findings.

https://wattsupwiththat.com/2017/12/30/editorial-narratives-in-science-journalism/n

In what follows I'm going to draw heavily on the work of Kevin McConway and David Spiegelhalter (1), two statisticians, who after getting tired of hearing bogus medical claims on the morning radio, developed a framework to assess the reporting of medical studies in the press. At points, I'm going to generalise to make the points apply beyond the medical sciences. My adapted framework contains eleven questions divided into two categories, study quality and the standard of reporting.

Scoring your article

All the questions in the framework can all be answered 'yes' or 'no' but you'll notice that they are sometimes worded in a quite ungainly fashion. This is because one point is awarded for a 'yes' and zero points are being awarded for a 'no'. Thus the higher an article's score the less trustworthy it is. An article with a score of seven or above should be considered deeply flawed, an article with 10 or more, utter bunkum.

Quality of study.

1. Just Observational? 

Have the researchers made any attempt to control for other variables or have they simply observed a process without interference? Whilst a lack of experimenter tampering may sound like a good thing, failing to apply proper controls make it extremely difficult to link a cause to an effect. Imagine testing a medical intervention but failing to control for other treatments. How can we tell which medical intervention caused an observed improvement?

2. Another Single study? 

What journalists often fail to realise is that scientific consensus cannot be built upon the outcome of one study. We should establish if the study in question has been successfully replicated, or if the results found reflect those found in other, similar investigations into the same phenomena. 

3. Might there be another explanation for the observed effect?

Is there a confounder that might explain the found results? The experimental controls should allow the researchers to eliminate plausible alternative explanations for an observed effect. Imagine experimenters are testing a new cold remedy. They select two groups, men and women. They give the women the new drug, but not the men. They find that the women in the group administered the medicine tend to recover more quickly than the men and report less extreme symptoms. They conclude the remedy is successful, but they have failed to control for gender. The experiment is confounded.

We should also consider any systematic bias, have the researchers introduced an element into the study that will skew the results in favour of one particular outcome? A striking example of this would be a recent survey issued by the Trump administration comparing voters opinions of the first year of Trump's first term to the first term of Obama's first term (below). You'll immediately notice that the first question has an element missing. Subjects are unable to rate Trump poorly whereas the option is available in the second question which asks subjects to rate Obama's first term(2). This quite laughable omission means that a side by side analysis is unsuitable.

4. Extrapolating Small sample sizes?

We should be extremely wary of studies with small sample sizes, especially those with subjects numbering in the tens rather than the hundreds or even thousands. There are mathematical ways to calculate appropriate sample sizes, but often it's easy enough to do this intuitively. You can't draw conclusions about millions of people based on a study of tens.  For example, consider Andrew Wakefield's withdrawn Lancet study which attempted to establish a link between the MMR vaccine and autism. Wakefield's study group contained five children, clearly not enough to draw conclusions about millions who had received the MMR vaccine. We need to be even more concerned when the conclusions of a series of tests are extrapolated to a much larger population

In relation to sample sizes, it's important to be wary of larger of studies of rare events. For example, a study of a rare illness may involve following millions of people but only an extremely small number of that sample develop the illness in question.

5. Samples not varied enough? 

Related to the previous point, it's not suitable to draw conclusions about a large population based on a sample that isn't varied enough. A good example of this is the study I looked at with the Spooktator crew early last year. The study proposed to show that individuals with strong religious or supernatural beliefs have poor cognitive abilities. The problem was, not only were the sample sizes extremely small but the vast majority of those studied were aged under 25 and female. It's not possible to draw conclusions about millions of believers of all ages and both sexes from such small, unvaried sample sizes. 

Standard of reporting.

6. Half (or less) of the story?

Are the reporters telling you everything? If they are reporting on the harmful effects of a medicine are they pointing out the benefits as well, or vice versa? Are they highlighting a small part of the research and ignoring the bigger picture? Researchers will normally point out flaws with their studies and suggest avenues for further research. Are these elements being covered in the report?

7. Representing risk in a misleading way? 

Watch out for the phrase "higher risk" in a report. If you are told that exposure to a substance doubles your risk of a certain ailment or illness it sounds quite bad. But what if your risk was incredibly low, to begin with? Unfortunately "X doubles the risk of Y" makes a fantastic attention-grabbing headline. This can also be true when considering a stated effect. If some variable makes the chance of a positive outcome more likely, we need to know how likely that outcome was in the first place to know if that is significant or not. To combat this we should be looking at absolute numbers as a sign of good science reporting.

8. An Exaggerated headline? 

Headlines for articles can be difficult to construct, this sometimes means important details are omitted, worse still they can be abandoned in favour of hyperbole. Does the headline of the article actually reflect what is said in the actual report, or is it misrepresentative or manipulated?

A great example of this would be a study published by the International Agency for Research on Cancer (IARC) reflecting the decision to list the radiation from mobile phones as “possibly carcinogenic to humans” or in specific terms to place classify it as a group 2B carcinogen (3). The 2B category is used when there is no specific evidence of a substance or material posing an actual risk, but there have been correlations made in the past. Some other 2B carcinogens include; fuels, laundry detergents and aloe vera.

The Daily Express clearly weren't interested in these details when they reported the IARC's report with the headline: "Shock Warning: Mobile phones can give you cancer" (4).

This headline complete strips the subtlety of the IARC report in favour of hyperbole and blind panic.

9. No Independent Comment?

When considering a scientific study it's vital to remember our first point, single studies do not make the scientific consensus. That means that we should be looking for independent comment from someone in the field of research not involved with the research in question to put our study in context. If an article omits this, it's likely based on promotional material issued by the institution that produced the research, one that has vested interest and may well not be as even-handed as one could hope. This doesn't mean these comments have to be negative, but they should be present.

This leads us to...

10. Does the report rely on public relations puff pieces, or are there considerable personal interests involved?

Are there elements of the report that imply the study is just PR? Who sponsored the research? What was the ultimate aim of the study? Does it fit into a wider scientific context? The answers to these questions are likely to tell you whether you should take the report with a pinch of salt or a shovel. This isn't to say that research that has been paid for by a company or corporation should be immediately disregarded, but it should be viewed with some skepticism. Likewise, research conducted by individuals with considerable personal interests in the research should be considered with suspicion.

For example, Martin Pall's research (5) on the dangers of electromagnetism should be weighted alongside the fact that he sells a range of supplements that he claims to strengthen the biological systems which his research claims EMF 'attacks'(6). Is this conflict of interest mentioned in the report, or in the original paper even?

.

11. Is the original research unavailable?

A great deal of the time, you'll find that the article you're reading doesn't even link to the original study. This means you're going to have to do the legwork yourself. The original study should be searchable by title if this gets you no results try searching by selected keywords. When doing your search, you may be far more likely to find success searching using an academic search programme such as Google Scholar (7). When you find your paper, it may well be hidden behind a paywall. Don't despair, even if this is the case, the abstract will be available for free. You will more likely than not find that this alone is sufficient to find errors in sloppy articles, especially if the author didn't even bother to read the abstract as often is the case!

Conclusion 

If you're slightly worried that all that may be difficult to remember, fear not, I've formed it into a handy mnemonic, JAMES H RANDI after my skeptical hero. You could always rework the framework to spell out the name of your own hero of science or skepticism. I've also formed the questions into a rudimentary scorecard which you can see below and download by following the link in the sources (8). Hopefully, it should make assessing science articles much easier.

Sources

(1) "Score and ignore: A radio listener's guide to ignoring health stories" McConway, Spiegelhalter, Significance, 2012. Accessed 17/12/17.

(2) https://slate.com/news-and-politics/2017/12/trump-invites-america-to-rate-his-and-obamas-presidencies-with-unimpeachably-evenhanded-survey.html

(3) http://www.iarc.fr/en/media-centre/pr/2011/pdfs/pr208_E.pdf

(4) https://www.express.co.uk/news/world/250043/Shock-Warning-Mobile-phones-can-give-you-cancer

(5) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3780531/

(6) http://www.nutricology.com/martin-pall-products/

(7) https://scholar.google.co.uk/

(8) https://drive.google.com/file/d/1s5CEQanT9JT5-fdL-skZ9ZcCXdyv_xI7/view?usp=sharing

Have Scientists REALLY Confirmed Life After Death Exists Or Has The Sun Created A New Level Of Hyperbole?

"LIGHT AT THE END OF THE TUNNEL: Scientists ‘confirm’ life after death exists with ground-breaking study" declares a breathless headline in Sunday's Sun newspaper. The story, unlike the paper's usual paranormal churn, doesn't reside in the paper's "weird" section, it sits proudly in the news proper section, although how much of an honour that is in this publication is questionable.

The first alarm bell that should sound upon reading the article is that the word "confirmed" appears in inverted commas. Did scientists confirm this or not? It turns out, reading further on in the article reveals that the words "life after death" should very probably also be in inverted commas, much like the approach the Telegraph took when reporting this story, way back in October 2014 (yep it's the same study from Southhampton). No-one ever said the broadsheets were above sensationalism, at least they use punctuation in a suitably weasely way.

So clearly when the Sun says "A new study shows people continue experiencing awareness for up to three minutes after death." what the reporter must actually mean is a study that's new to them, as this study, by Dr Sam Parnia and Southampton University was published in Resuscitation: The Journal of The European Resuscitation Council back in their December 2014 volume!
 
The article begins with what I believe passes for philosophy and deep thought in the Murdoch press:

"DEATH is an inevitable consequence of life, but scientists believe they may have found some light at the end of the tunnel. Life after death has been “confirmed” by experts who say consciousness continues even once a person’s heart has stopped beating."

So far so good, but it soon becomes apparent that this isn't the kind of "life after death" you may have been expecting.

"In a study of more than 2,000 people, British scientists confirmed that thought persists after death, and simultaneously uncovered convincing evidence of an out-of-body experience for a patient declared dead by medics. Scientists had believed the brain ceased all activity 30 seconds after the heart stopped pumping blood around the body, and that awareness stopped at the same time.But research from the University of Southampton suggests otherwise.A new study shows people continue experiencing awareness for up to three minutes after death."

Three minutes that's hardly skipping through the clouds with Aunt Mildred as that headline may have led you to believe, nor is this a newly discovered phenomena. All that has changed is the length of time consciousness has been found continued after the heart stopped beating, if the study is correct. This also highlights a common failing in reports about "death", there are different standards of death. What the study is referring to is the heart stopping and the cessation of respiration, clinical death.  This isn't death as such. The success of modern resuscitation methods has necessitated the introduction of the concept of "brain death" when the brain stem dies and there is no further hope of resuscitation.

Obviously, it isn't this form of death which these patients experienced. Had it been, I suspect that Dr Parnia and his colleagues may have had some trouble surveying them about their cognitive experiences whilst their heart had stopped. Here's the critical flaw in reporting of the significance of this study: none of these patients died. This isn't a study of consciousness after "death" or extrapolated further "life after death". It's a study of consciousness during heart failure. You may well think that it's superfluous for me to point out this blatantly obvious flaw, which really should go without saying. To that I respond, the Sun ran with this story a stock image (below) of a body with a bloody toe-tag on it! I don't think the reporter who wrote this considered this distinction at all when they lazily picked images to fit in the text!

Laughably the Sun then quote Dr Sam Parnia, the head researcher on the study, who gives a definition of "death" which completely discredits all of the Sun's hyperbole thus far:

“Contrary to perception, death is not a specific moment but a potentially reversible process that occurs after any severe illness or accident causes the heart, lungs and brain to cease functioning.If attempts are made to reverse this process, it is referred to as ‘cardiac arrest’; however, if these attempts do not succeed it is called ‘death’.”

So Parnia believes that "clinical death" is a somewhat moot term, that's arguable, but it's clear by that standard he cannot consider any of the patients surveyed to have actually experienced death!












Let me further put the boot into this article, the study doesn't mention clinical death once. Patients are described as cardiac arrest sufferers. The addition of ideas of "death" have clearly been added to the press coverage for purposes of sensationalism. Now, it's perhaps unfair to ascribe this to the Sun, as it's clearly an aspect of the earlier Telegraph report. What worries me is that no one has corrected the press with regards to this blatant inaccuracy, or someone has and the press have completely ignored their words of caution. I can only speculate on that, but what I'm sure of is no one from the Sun bothered to check the paper they were reporting on, and they have no urge to allow their readers to do the same as they don't actually link to the paper in the story. Clicking the link highlighted by the word study takes the reader to a laughable page of all the "research" the Sun has reported on (left).


At best the only finding we can extrapolate from the original study, and that's if we consider it methodologically sound, is that consciousness may survive for a short time after the heart ceases, perhaps longer than initially suspected. Even this finding of the possible extended duration of consciousness is perhaps not surprising, we may well be discovering this now as a result of modern resuscitation methods are bringing patients "back" from clinical deaths after longer periods.

Ah but wait... there is something extraordinary left to be explored here, remember this tease from earlier in the article?

"...(the study) simultaneously uncovered convincing evidence of an out-of-body experience for a patient declared dead by medics...."

To be clear this is an abnormality of one patient in 2000, in terms of the study there is nothing statistically significant here. Also, I don't see any reason to suspect that this was an out-of-body experience or to suggest it as evidence of mind/body duality. Here's how Parnia describes the patient's experience:

“This is significant, since it has often been assumed that experiences in relation to death are likely hallucinations or illusions occurring either before the heart stops or after the heart has been successfully restarted, but not an experience corresponding with ‘real’ events when the heart isn’t beating. In this case, consciousness and awareness appeared to occur during a three-minute period when there was no heartbeat.This is paradoxical, since the brain typically ceases functioning within 20-30 seconds of the heart stopping and doesn’t resume again until the heart has been restarted. Furthermore, the detailed recollections of visual awareness in this case were consistent with verified events.”

Parnia should know better, what he is describing here is pure anecdote. He only has testimonies of the patient and those involved in the resuciation that the events were similar, also we are given no indication of how similar. I'm pretty sure I could roughly describe what's going on in an emergency room, how specific was the patient. Secondly, maybe he could hear what was going on around him, we've established that he may have had consciousness, is it that much of a stretch to imagine he had auditory input too? Am I missing something, and if I am, please point it out to me, why does any of this suggest an out of body experience?

Perhaps unsurprisingly there's no mention of this particular patient in the abstract, methodology, results or conclusion of Parnia's paper. It may lie somewhere in the main body of text unavailable to me as it's behind a paywall, but one should expect if it had the significance Parnia ascribes to it, one would find it in these sections.

It makes me rather sad to see a legitimate study, though I suspect quite flawed, reported in such a sensationalist way. Especially considering it seems the head researcher has been happy to indulge such sensationalism. There's an interesting and important hypothesis lurking in this paper, and it may well yield findings and further studies to match. It's a shame that a public becoming more open to scientific reporting is unlikely to ever be exposed to these things.