Ben Goldacre’s Bad Evidence

Ben Goldacre

Ben Goldacre’s interesting programme on evidence-based policy making went out yesterday evening. Like so much of his work, I found myself alternately agreeing vigorously and disagreeing in exasperation. The trouble is not what he does say, but what he doesn’t.

His central argument is a familiar one.

In medicine, scientists determine what works using randomised controlled trials. Give one set of patients a pill, give another set a placebo, and see what difference the pill makes. Do this lots of times, trying to control for confounding variables (like the participants’ lifestyles) and if possible make it “double blind” by ensuring neither the participants nor the researchers conducting the test know which group anyone is in.

This method gives us a high degree of certainty that some pills work while others don’t, or do so less well. It is far superior to simply acting on a hunch, monitoring a particular outcome and then assuming it was a result of your pill, without checking whether it might have been some other variable you haven’t considered and controlled.

So why can’t this method be applied to public policy? Why do we subject children to educational methods, the unemployed to work programmes, and criminals to rehabilitation methods that lack this rigorous evidence?

It’s a good question, and I completely agree with him that it would be good to do this more to weed out the “bad evidence” that informs so many policies. I’ve been supporting Jenny Jones in her scrutiny of the Mayor of London’s mandatory work experience pilot, which seems bedevilled by bad evidence, never mind whether it might be wasting the time of young people suffering in a very difficult jobs market and now feeling punished for it.

I felt sufficiently strongly about our approach to policy making to include several proposals in the Green Party’s manifesto for the 2012 London Mayoral and Assembly elections (PDF).

But there is also a danger that evidence arrived at through rigorous research could become “bad evidence” if it were applied technocratically.

For what Goldacre’s radio programme ignored, inexplicably, is the normative element of policy. He talked about “outcomes”, but how do we define a good outcome? It might seem obvious – stop a criminal reoffending, get a young person back into work. But it isn’t that simple.

It might be the case that one particular approach to criminal justice is more effective than another, but it might be considered unjust. What if we found that all forms of punishment led to higher reoffending rates? Should we drop our long-held belief in the moral right of punishment in favour of better “outcomes”? This is a normative, moral question – short of brushing it aside we cannot ignore the role of normative considerations.

Both the present and previous governments went for “workfare” schemes where unemployed people lose their benefits if they refuse to take up work placements. One of the supposed outcomes of this policy is that more people get work as a result. My reading  of the evidence for these suggests they don’t. But proponents also make a normative claim that it is right to make people work for their benefits, especially if they haven’t worked for a long time, if at all. On the other side, some (myself included) think that a compassionate and wealthy society such as ours can extend a universal right to a basic standard of living and shouldn’t impose conditions on those basic benefits.

Goldacre didn’t say that evidence should trump political philosophy. But the two can often get confused in political debate. Politicians can lose the courage of their convictions and feel compelled to assert that evidence supports their case, when they began sure only of their convictions. Opposition can often feel that a policy must be “wrong” because the evidence shows it doesn’t achieve the outcome they would think right, perhaps ignoring the different view of a “right outcome” held by the Government.

It isn’t sufficient to consider these points in isolation – to, on the one hand, ascertain the evidence about the outcomes of a particular policy, and on the other to have one’s normative beliefs entirely in parallel, and to then attempt to reconcile (or more likely confuse) them in the murky world of political debate. Normative and empirical considerations must inform each other.

Another very interesting series on Radio 4, Melvyn Bragg’s examination of “the value of culture“, included a very good programme today on the old “two cultures” debate most famously expressed by C. P. Snow. He was concerned in the 1950s that artists weren’t assimilating the advances of science, and vice versa, to the detriment of both, and to the point where both “parallel cultures” viewed each other with suspicion. Instead of seeking to bridge the gap through understanding and engagement, they preached at each other. Goldacre’s programme  would have been much improved if he had engaged with both the empirical and the normative.


  1. Ben Goldacre said:

    “It might be the case that one particular approach to criminal justice is more effective than another, but it might be considered unjust.”

    It would be completely senseless to do a randomised trial of an intervention that was socially or practically unacceptable, because it would be interrogating a completely irrelevant question, the effectiveness of something that is not going to be used.

    Nobody has ever suggested that randomised trials can tell you whether an intervention is morally right. But they are extremely good to tell you if they achieve their stated objectives, and more need to be done.

    Saying “more of X needs to be done” does not mean “only X needs to be done”, as was made very clear in the programme.

    2nd January 2013
    • Tom Chance said:

      Hi Ben, thanks for leaving a comment.

      Perhaps my example of a justice system without punishment was a bit extreme, and ‘unjust’ a stark term. In the policy areas you discussed in your programme there are many possible interventions where the morality and other normative aspects are bundled up with empirical aspects, and the political reasons (noble and mucky) make the evaluation of all those aspects rather difficult to disentangle.

      My criticism wasn’t that you are wrong to advocate RCTs, or that you had incorrectly suggested they can replace normative considerations. I just felt it was a shame you didn’t explore policy questions in the round, looking at how RCTs might feed into a contentious political process. Your presentation came across as technocratic.

      Just as it is blinkered to enact policies for moral reasons without empirical evidence to suggest the desired outcomes will be achieved (if indeed there are any consequences in question), so it would be blinkered to enact or retract policies based on RCTs about outcomes without a proper consideration of the normative reasons for the policy. I’m sure many Tories would think workfare is the Right Thing To Do, irrespective of whether it is effective in getting people back into work.

      Admittedly most political commentary focuses on the normative side, with a few outfits like FullFact making great inroads in an under-valued territory, so perhaps in a short radio programme it’s fine just to examine and advocate one empirical method. I don’t think it is.

      2nd January 2013

Comments are closed.