Ben Goldacre’s Bad Evidence

Ben Goldacre’s interesting programme on evidence-based policy making went out yesterday evening. Like so much of his work, I found myself alternately agreeing vigorously and disagreeing in exasperation. The trouble is not what he does say, but what he doesn’t.

His central argument is a familiar one.

In medicine, scientists determine what works using randomised controlled trials. Give one set of patients a pill, give another set a placebo, and see what difference the pill makes. Do this lots of times, trying to control for confounding variables (like the participants’ lifestyles) and if possible make it “double blind” by ensuring neither the participants nor the researchers conducting the test know which group anyone is in.

This method gives us a high degree of certainty that some pills work while others don’t, or do so less well. It is far superior to simply acting on a hunch, monitoring a particular outcome and then assuming it was a result of your pill, without checking whether it might have been some other variable you haven’t considered and controlled.

So why can’t this method be applied to public policy? Why do we subject children to educational methods, the unemployed to work programmes, and criminals to rehabilitation methods that lack this rigorous evidence?

It’s a good question, and I completely agree with him that it would be good to do this more to weed out the “bad evidence” that informs so many policies. I’ve been supporting Jenny Jones in her scrutiny of the Mayor of London’s mandatory work experience pilot, which seems bedevilled by bad evidence, never mind whether it might be wasting the time of young people suffering in a very difficult jobs market and now feeling punished for it.

I felt sufficiently strongly about our approach to policy making to include several proposals in the Green Party’s manifesto for the 2012 London Mayoral and Assembly elections (PDF).

But there is also a danger that evidence arrived at through rigorous research could become “bad evidence” if it were applied technocratically.

For what Goldacre’s radio programme ignored, inexplicably, is the normative element of policy. He talked about “outcomes”, but how do we define a good outcome? It might seem obvious – stop a criminal reoffending, get a young person back into work. But it isn’t that simple.

It might be the case that one particular approach to criminal justice is more effective than another, but it might be considered unjust. What if we found that all forms of punishment led to higher reoffending rates? Should we drop our long-held belief in the moral right of punishment in favour of better “outcomes”? This is a normative, moral question – short of brushing it aside we cannot ignore the role of normative considerations.

Both the present and previous governments went for “workfare” schemes where unemployed people lose their benefits if they refuse to take up work placements. One of the supposed outcomes of this policy is that more people get work as a result. My reading  of the evidence for these suggests they don’t. But proponents also make a normative claim that it is right to make people work for their benefits, especially if they haven’t worked for a long time, if at all. On the other side, some (myself included) think that a compassionate and wealthy society such as ours can extend a universal right to a basic standard of living and shouldn’t impose conditions on those basic benefits.

Goldacre didn’t say that evidence should trump political philosophy. But the two can often get confused in political debate. Politicians can lose the courage of their convictions and feel compelled to assert that evidence supports their case, when they began sure only of their convictions. Opposition can often feel that a policy must be “wrong” because the evidence shows it doesn’t achieve the outcome they would think right, perhaps ignoring the different view of a “right outcome” held by the Government.

It isn’t sufficient to consider these points in isolation – to, on the one hand, ascertain the evidence about the outcomes of a particular policy, and on the other to have one’s normative beliefs entirely in parallel, and to then attempt to reconcile (or more likely confuse) them in the murky world of political debate. Normative and empirical considerations must inform each other.

Another very interesting series on Radio 4, Melvyn Bragg’s examination of “the value of culture“, included a very good programme today on the old “two cultures” debate most famously expressed by C. P. Snow. He was concerned in the 1950s that artists weren’t assimilating the advances of science, and vice versa, to the detriment of both, and to the point where both “parallel cultures” viewed each other with suspicion. Instead of seeking to bridge the gap through understanding and engagement, they preached at each other. Goldacre’s programme  would have been much improved if he had engaged with both the empirical and the normative.