I Think You'll Find It's a Bit More Complicated Than That (3 page)

If a scientist sidesteps their scientific peers, and chooses to take an apparently changeable, frightening and technical scientific case directly to the public, then that is a deliberate decision, and one that can’t realistically go unnoticed. The lay public might find your case superficially appealing, but they may not be fully able to judge the merits of all your technical evidence.

I think these serious scientific concerns belong, at least once, in a clear scientific paper. I don’t see how this suggestion is inappropriate, or impudent, and in all seriousness, I can’t see an argument against it. I hope it won’t elicit an accusation of sexism, or of participation in a cover-up. I hope that it will simply result in an Oxford science professor writing a scientific paper, about a scientific claim of great public health importance, that she has made repeatedly – but confusingly – for at least half a decade.

Cherry-Picking
Is Bad. At Least Warn Us When You Do It

Guardian
, 24 September 2011

Last week the
Daily Mail
and Radio 4’s
Today
programme took some bait from
Aric Sigman
, an author of popular-sciencey books about the merits of traditional values. ‘Sending babies and toddlers to daycare could do untold damage to the development of their brains and their future health,’ explained the
Mail
.

These news stories were based on a scientific paper by Sigman in the
Biologist
. It misrepresents individual studies, as
Professor Dorothy Bishop
demonstrated almost immediately, and it cherry-picks the scientific literature, selectively referencing only the studies that support Sigman’s view. Normally this charge of cherry-picking would take a column of effort to prove, but this time
Sigman himself admits it
, frankly, in a
PDF posted on his own website
.

Let me explain why this behaviour is a problem. Nobody reading the
Biologist
, or its press release, could possibly have known that the evidence presented was deliberately incomplete. That is, in my opinion, an act of deceit by the journal; but it also illustrates one of the most important principles in science, and one of the most bafflingly recent to emerge.

Here is the paradox. In science, we design every individual experiment as cleanly as possible. In a trial comparing two pills, for example, we make sure that participants don’t know which pill they’re getting, so that their expectations don’t change the symptoms they report. We design experiments carefully like this to exclude bias: to isolate individual factors, and ensure that the findings we get really do reflect the thing we’re trying to measure.

But individual experiments are not the end of the story. There is a second, crucial process in science, which is synthesising that evidence together to create a coherent picture.

In the very recent past, this was done badly. In the 1980s, researchers such as
Celia Mulrow
produced damning research showing that review articles in academic journals and textbooks, which everyone had trusted, actually presented a distorted and unrepresentative view when compared with a systematic search of the academic literature. After struggling to exclude bias from every individual study, doctors and academics would then synthesise that evidence together with frightening arbitrariness.

The science of
‘systematic reviews’
that grew from this research is exactly that: a science. It’s a series of reproducible methods for searching information, to ensure that your evidence synthesis is as free from bias as your individual experiments. You describe not just what you found, but how you looked, which research databases you used, what search terms you typed, and so on. This apparently obvious manoeuvre has revolutionised the science of medicine.

What does that have to do with Aric Sigman, the Society of Biologists, and their journal, the
Biologist
? Well, this article was not a systematic review, the cleanest form of research summary, and it was not presented as one. But it also wasn’t a reasonable summary of the research literature, and that wasn’t just a function of Sigman’s unconscious desire to make a case: it was entirely deliberate. A deliberately incomplete view of the literature, as I hope I’ve explained, isn’t a neutral or marginal failure. It is exactly as bad as a deliberately flawed experiment, and to present it to readers without warning is bizarre.

Blame is not interesting, but I got in touch with the Society of Biology, as I think we’re more entitled to have high expectations of them than of Sigman, who is, after all, some guy writing fun books in Brighton. They agree that what they did was wrong, that mistakes were made, and that they will do differently in future.

Here’s why I don’t think that’s true.
The last time
they did exactly the same thing, not long ago, with another
deliberately incomplete article
from Sigman, I wrote to the journal, the editor, and the editorial board, setting out these concerns very clearly.

The
Biologist
has actively decided to continue publishing these pieces by Sigman, without warning. They get the journal huge publicity; and fair enough. I’m no policeman. But in the two-actor process of communication, until it explains to its readers that it knowingly presents cherry-picked papers without warning – and makes a public commitment to stop – it’s for readers to decide whether they can trust what the journal publishes.

Being Wrong

Guardian
, 15 July 2011

Morons often like to claim that their truth has been suppressed: that they are like Galileo, a noble outsider fighting the rigid and political domain of the scientific literature, which resists every challenge to orthodoxy.

Like many claims, this is something for which it’s possible to gather data.

Firstly, there are individual anecdotes that demonstrate the routine humdrum of medical fact being overturned. We used to think that hormone-replacement therapy significantly reduced the risk of heart attacks, for example, because this was the finding of several large observational studies. That kind of research has important limitations: if you just grab some women who are receiving prescriptions for HRT from their doctors, and compare them to women who aren’t getting HRT, you might well find that the women on HRT are healthier, but that might simply be because they were healthier to start with. Women on HRT might be richer, or more interested in their health, for example. At the time, this research represented our best guess, and that’s often all you have to work with. Eventually, after decades of HRT being widely used, a large randomised trial was conducted: they took 16,000 women who were eligible for HRT, and randomly assigned them to receive either real hormones, or a dummy placebo pill. At long last we had a fair test, and after several years of treatment had passed, in 2002, the answer fell out. HRT increased the risk of having a heart attack by 29 per cent.

Were these findings suppressed? No. They were greeted eagerly, and with some horror: in fact, the finding was so concerning that the trial had to be stopped early, to avoid putting any further participants at risk, and medical practice was overturned.

Even the supposed stories of outright medical intransigence turn out, on close examination, to be pretty weak: people claim that doctors were slow to embrace
Helicobacter pylori
as the cause of gastric ulcers, when in reality it only took a decade from the first murmur of a research finding to international guidelines recommending antibiotic treatment for all patients with ulcers.

But individual stories aren’t enough. This week Vinay Prasad and colleagues published a fascinating piece of
research about research
. They took all 212 academic papers published in the
New England Journal of Medicine
during 2009. Of those, 124 made some kind of claim about whether a treatment worked or not. Then, they set about measuring how those findings fitted into what was already known. Two reviewers assessed whether the results in each study were positive or negative, and finally – separately – they decided whether these new findings overturned previous research.

Seventy-three of the studies looked at new treatments, so there was nothing to overturn. But the remaining fifty-one were very interesting, because they were, essentially, evenly split: sixteen upheld a current practice as beneficial; nineteen were inconclusive; and, crucially, sixteen found that a practice believed to be effective was in fact ineffective, or vice versa.

Is this unexpected? Not at all. If you like, you can look at the same problem from the opposite end of the telescope. In 2005,
John Ioannidis
gathered together all the major clinical research papers published in three prominent medical journals between 1990 and 2003; specifically, he took the ‘citation classics’, the forty-nine studies that were cited more than a thousand times by subsequent academic papers.

Then he checked to see whether their findings had stood the test of time, by conducting a systematic search in the literature to make sure he was consistent in finding subsequent data. Of his forty-nine citation classics, forty-five had found that an intervention was effective, but in the time that had passed subsequently, only half of these findings had been positively replicated. Seven studies – 16 per cent of the total – were flatly contradicted by subsequent research; and for a further seven studies, follow-up research had found that the benefits originally identified were present, but more modest than first thought.

This looks like a reasonably healthy state of affairs to me: there probably are true tales of dodgy peer reviewers delaying publication of findings they don’t like, but overall, things are routinely proven to be wrong in academic journals. Equally, the other side of this coin is not to be neglected: we often turn out to be wrong, even with giant, classic papers. So it pays to be cautious with dramatic new findings; if you blink you might miss a refutation; and there’s never an excuse to stop monitoring outcomes.

Kids Who Spot
Bullshit, and the Adults Who Get Upset About It

Guardian
, 28 May 2011

If you can tear yourself away from
Ryan Giggs’s penis
for just one moment, I have a different censorship story.

Brain Gym
is a schools programme I’ve been
writing about since 2003
. It’s a series of elaborate physical movements with silly pseudoscientific justifications: you wiggle your head back and forth, because that gets more blood into your frontal lobes for clearer thinking; you contort your fingers together to improve some unnamed ‘energy flow’. They’re keen on drinking water, because ‘processed foods’ – I’m quoting the Brain Gym Teacher’s Manual – ‘do not contain water’. You
pay hundreds of thousands
of pounds for Brain Gym, and it’s still done in
hundreds of state schools
across the UK.

This week I got an email from a science teacher about a thirteen-year-old pupil. Both have to remain anonymous. This pupil wrote an article about Brain Gym for her school paper, explaining why it’s nonsense: the essay is respectful, straightforward, and factual. But the school decided this article couldn’t be printed, because it would offend the teachers in the junior school who use Brain Gym.

Now, this is weak-minded, and perhaps even vicious. More interesting, though, is how often children are able to spot bullshit, and how often adults want to shut them up.

Emily Rosa
is the youngest person ever to have
published a scientific paper
in the
Journal of the American Medical Association
, one of the most influential medical journals in the world. At the age of nine she saw a TV programme about nurses who practise ‘Therapeutic Touch’, claiming they can detect and manipulate a ‘human energy field’ by hovering their hands above a patient.

For her school science-fair project, Rosa conceived and executed an experiment to test if they really could detect this ‘field’. Twenty-one experienced practitioners put their palms on a table, behind a screen. Rosa flipped a coin, hovered her hand over the therapist’s left or right palm accordingly, and waited for them to say which it was: the therapists performed no better than chance, and with 280 attempts there was sufficient statistical power to show that these claims were bunk.

Therapeutic Touch practitioners, including some in university posts,
were deeply unhappy
: they insisted loudly that
JAMA
was wrong to publish the study.

Closer to home is
Rhys Morgan
, a schoolboy with Crohns Disease. Last year, chatting on
www.crohnsforum.com
, he saw people recommending ‘Miracle Mineral Solution’, which turned out to be industrial bleach, sold with a dreary conspiracy theory to cure Aids, cancer, and so on.

At the age of fifteen, he was perfectly capable of exploring the evidence,
finding official documents
, and explaining why it was dangerous.
The adults banned him
. Since then he’s got his story on
The One Show
, while the
Chief Medical Officer for Wales
, the Food Standards Agency and Trading Standards
have waded in
to support him.

People wring their hands over how to make science relevant and accessible, but newspapers hand us one answer on a plate every week, with their barrage of claims on what’s good for you or bad for you: it’s
evidence-based medicine
. If every school taught the basics – randomised trials, blinding, cohort studies, and why systematic reviews are better than cherry-picking your evidence – it would help everyone navigate the world, and learn some of the most important ideas in the whole of science.

But even before that happens, we can feel optimistic. Information is more easily accessible now than ever before, and smart, motivated people can sidestep traditional routes to obtain knowledge and disseminate it. A child can know more about evidence than their peers, and more than adults, and more than their own teachers; they can
tell the world
what they know, and they can have an impact.