March 9, 2018, by Brigitte Nerlich

If not evidence-based, then what?

In my last post on science communication, I quoted from an article by David Dickson in which he said that “evidence-based decision-making is an ideal that we should aspire to at every level of society, from local communities to the top levels of government”.

Evidence – a twitter discussion

There was a bit of back-lash against this from the STS community. Silvio Funtowitz said that this sentence confirmed Clark Miller’s suspicion (earlier expressed on twitter) that science communication was a PR wing of science as a multi-billion dollar business. I asked Silvio why he thought that and he said: “It seems that the only purveyors of privileged ‘evidence’ are the scientists. Not to mention the fact that ‘evidence’ is vague and ambiguous.” We then discussed definitions of evidence. Silvio asked for an ‘operational definition’ and I said: “Information that increases the likelihood that something is true”. I’d love to hear people’s views on that – as I only came up with that on the spur of the twitter moment!

Another twitter conversation happened between me and Andrea Saltelli who kindly sent me various articles in which researchers like himself show what’s wrong with evidence-based policy. Unfortunately, they went a bit over my head, but they made me think and search for more articles on this topic. Overall, I think there is some discomfort with quantitative modelling or the privileging of it and a proposal is made to replace this with ‘quantitative story-telling’. His last tweet said: “Evidence based policy is a political concept, not a scientific one. Among its functions is that of legitimizing the existing power asymmetries in the production of evidence.” That really made me think.

I agree that evidence can be used politically in the context of policy. Indeed, this is almost a truism. I did not understand why it was not a scientific concept or why that had to be emphasised. Again, I thought, of course, we are dealing with a concept that is embedded in the context of policy making. So, by definition it’s not a scientific concept. As I found out, I might actually have been wrong in this assumption. I also agree that the concept of evidence can be used to legitimise existing power asymmetries, but so can any other concept used in politics and policy. Again, that is almost a truism. That does not mean, I mused, that one should throw out the concept or indeed, as Dickson said above, the ‘ideal’ of evidence-based policy or decision-making with the bath-water of critique.

(I want to stress that I am no expert on evidence and there is a LOT of literature on the topic. So please chip in with comments pointing out where I have gone wrong and where better or more info can be found. My colleagues Andrew Turner,  Warren Pearce and Sujatha Raman in particular know much more about this than I do)

Some history – evidence-based medicine

As is my want, I began to rummage around a bit in the history of the concept. As everybody knows, the concept of evidence-based policy has its roots in the concept of evidence-based medicine. To be more precise: “In the spring of 1990 the young McMasters University Internal Medicine residency coordinator, Dr. Gordon Guyatt, had just introduced a new concept he called ‘Scientific Medicine.’ The term described a novel method of teaching medicine at the bedside. It was built on groundwork laid by his mentor Dr. David Sackett, using critical appraisal techniques applicable to the bedside. However, the response from his fellow staff was anything but warm and inviting. The implication that current clinical decisions were less than scientific, although probably true, was nonetheless unacceptable to them. Guyatt then returned with a new title that described the core curriculum of the residency program: ‘Evidence-Based Medicine’ (EBM). The coined term appeared in a subsequent 1991 ACP Journal Club editorial.”

That was interesting, as it showed that ‘evidence-based’ medicine was indeed a ‘scientific’ concept, but that, like in the later case of ‘evidence-based’ policy the response from people who were introduced to that concept was “anything but warm and inviting”, although for different reasons. In the case of evidence-based medicine, those supposed to practice it resented the fact that their traditional ways of doing and knowing were deemed to be “less than scientific”, while in the case of evidence-based policy, those advising people on the policy-making process warned them that this might lead to policy-making based on nothing but science.

According to oral history, the concept of evidence-based medicine was often misunderstood. One founder pointed out that in contrast to what many believed: “it wasn’t old hat, impossible to practice, cookbook medicine, the creature of managers and purchasers, or concerned only with randomized trials.” Some of these misunderstandings seem to have carried over into current critiques of evidence-based policy-making.

Interestingly, again according to oral history, evidence-based medicine challenged ‘expert-based medicine’ and thus empowered young physicians, nurses and others to challenge authority and dogma. That’s something rather positive and it was picked up by policy makers.

Some history – evidence-based policy

Overall then, evidence-based medicine began to flourish in the 1990s. It seems that policy makers almost immediately wanted to emulate these efforts at making the practice of medicine ‘better’ in order to make the practice of policy ‘better’, that is to say, to engage in policy-making that can be questioned and improved, because it is based on evidence rather than ideology and dogma.

As Wikipedia points out: “Underlying many of the calls for ‘evidence based policy’ is often a (stated or unstated) concern with fidelity to scientific good practice, reflecting the belief that social goals are best served when scientific evidence is used rigorously and comprehensively to inform decisions, rather than in a piecemeal, manipulated, or cherry-picked manner.”

The term evidence-based policy seems to have first been used in 1996 by the statistician “Adrian Smith in his 1996 presidential address to the Royal Statistical Society, [in which] Smith questioned the current process of policy making and urged for a more ‘evidence-based approach’ commenting that it has ‘valuable lessons to offer’.” (Wikipedia) It was then taken up by the Labour party. Now it’s a phrase that has general, albeit increasingly contested, political currency.

Evidence-based policy – what is it, what should it be?

When looking at some introductory books on the matter of evidence-based policy, I found two which looked interesting – but there are hundreds I suppose. One, by Ray Pawson, is from 2006 and entitled Evidence-based Policy Making: A realist perspective, the other is by Justin Parkhurst entitled The Politics of Evidence: From evidence-based policy to the good governance of evidence. The first is worth a read and really quite fun. I’ll focus here on the second published in 2017, as it deals with more recent debates that inform the tweets I mentioned at the beginning. Here are a few salient points made in the book.

First, the books makes clear that given the “fundamentally contested nature of most public policy concerns, the use of evidence for policy has been described as ‘qualitatively different’ (Black 2001) than its use in technical decision-making arenas (such as clinical medicine)” (Parkhurst, p. 5).

Second, one should not only distinguish between the use of evidence in technical decision-making and the use of evidence in public policy decision-making, but also between the use of evidence in public policy-making and the ‘good governance of evidence’ in the public policy-making process.

Third, such governance of evidence-use in policy relies on spotting and dealing with two types of biases; what Parkhurst calls ‘technical bias’ and ‘issue bias’. According to Parkhurst, so-called evidence advocates are concerned that “political interests drive the misuse or manipulation of evidence” – leading to technical bias -,  while critical policy scholars highlight that “appeals to ‘evidence-based policy’ can depoliticise political debates” (see Parkhurst, p. 6) – leading to issue bias. “The first form of bias broadly reflects the value of scientific fidelity, while the second broadly reflects the value of democratic representation.”

The governance of evidence in public policy-making

What’s the way forward – how can evidence-based policy be governed in such a way that these biases are avoided or, indeed, these values reconciled?

Parkhurst has a proposal: “efforts to improve evidence use will ultimately require building systems that work to embed key normative principles about evidence utilisation into policy processes – systems that can be said to govern the use of evidence within policymaking. Therefore, in order to move the EBP field forward, it is necessary to consider how to establish evidence advisory systems that promote the good governance of evidence – working to ensure that rigorous, systematic and technically valid pieces of evidence are used within decision-making processes that are inclusive of, representative of and accountable to the multiple social interests of the population served.” (p. 8)

To this I would add: the good governance of evidence project should extend its reach from the use of evidence in policy-making to the collection of evidence in science, i.e to ‘the good governance of science’. At the moment the governance of science is in crisis, as a regime of perverse incentives is rolled out, such as REF and impact, and universities and other public agencies are becoming commodified, industrialised, marketised and financialised. This compromises the gathering and value of scientific evidence. Another danger is posed by the entanglement of science and politics fostered by these developments.

Evidence is a tool, a technology; it can be used for good or ill. It has to be collected and used with care. This includes creating the right conditions for scientists to do their job.

Conclusion

I think it is important to make a distinction between (scientific) evidence and the use of (scientific) evidence in policy and politics – which can lead to the political use of that evidence. In my view, various types of scientific evidence (from quantitative modelling to qualitative case-studies) are a necessary but not sufficient conditions for devising policies that deal with health, medicine, technology, innovation, climate change (mitigation, adaptation) and so on. Scientific evidence is a key component of modern policy analysis and policy decisions, but it is, of course, always used alongside and traded against other components, such as political and social values.

Scientific evidence gathered and used by policy makers is always partial, limited and contested; it can come from reliable sources or unreliable ones; it can come from a narrow range or plurality of sources or a broad one; it can be used and misused in the pursuit of power; it can be dismissed or it can be given unwarranted priority. There can be “cherry-picking, obfuscation or manipulation” (Parkhurst, 2017: 4) All these things can happen. What always does happen however is that scientific evidence is interpreted by policy makers in light of pre-existing knowledge and pre-judices. After that policy makers have to use their power of judgement and their political power to come to a decision – their decision.

But I don’t see how policies that affect how we live, work and die in modern high-tech societies can do without scientific evidence or input, i.e. can NOT be evidence-based or, to use a weaker form of words, can NOT be evidence-informed. The political processes of evidence-gathering and evidence-use (including the gathering and use of scientific evidence) should, of course, be governed by shared normative principles, as outlined by Parkhurst. They should also be informed by the ideal of evidence-based policy.

And finally, scientific evidence ALONE, indeed any type of evidence alone (see Parkhurst, p. 5) will never sway policy makers and neither should it. This seems to be quite a preposterous proposition to me. We should certainly not dismiss the notion of evidence-based policy based on the assumption that this has ever been proposed. Has it? I’d love to know.

PS. I had just finished writing this when I saw this tweet which is a direct counterpoint to what I have said here. Discuss!

Image: Wikimedia commons

• Share on Facebook
• Share on Twitter
• Share on Google+
• Share on LinkedIn