March 9, 2018, by Brigitte Nerlich
If not evidence-based, then what?
In my last post on science communication, I quoted from an article by David Dickson in which he said that “evidence-based decision-making is an ideal that we should aspire to at every level of society, from local communities to the top levels of government”.
Evidence – a twitter discussion
There was a bit of back-lash against this from the STS community. Silvio Funtowitz said that this sentence confirmed Clark Miller’s suspicion (earlier expressed on twitter) that science communication was a PR wing of science as a multi-billion dollar business. I asked Silvio why he thought that and he said: “It seems that the only purveyors of privileged ‘evidence’ are the scientists. Not to mention the fact that ‘evidence’ is vague and ambiguous.” We then discussed definitions of evidence. Silvio asked for an ‘operational definition’ and I said: “Information that increases the likelihood that something is true”. I’d love to hear people’s views on that – as I only came up with that on the spur of the twitter moment!
Another twitter conversation happened between me and Andrea Saltelli who kindly sent me various articles in which researchers like himself show what’s wrong with evidence-based policy. Unfortunately, they went a bit over my head, but they made me think and search for more articles on this topic. Overall, I think there is some discomfort with quantitative modelling or the privileging of it and a proposal is made to replace this with ‘quantitative story-telling’. His last tweet said: “Evidence based policy is a political concept, not a scientific one. Among its functions is that of legitimizing the existing power asymmetries in the production of evidence.” That really made me think.
I agree that evidence can be used politically in the context of policy. Indeed, this is almost a truism. I did not understand why it was not a scientific concept or why that had to be emphasised. Again, I thought, of course, we are dealing with a concept that is embedded in the context of policy making. So, by definition it’s not a scientific concept. As I found out, I might actually have been wrong in this assumption. I also agree that the concept of evidence can be used to legitimise existing power asymmetries, but so can any other concept used in politics and policy. Again, that is almost a truism. That does not mean, I mused, that one should throw out the concept or indeed, as Dickson said above, the ‘ideal’ of evidence-based policy or decision-making with the bath-water of critique.
(I want to stress that I am no expert on evidence and there is a LOT of literature on the topic. So please chip in with comments pointing out where I have gone wrong and where better or more info can be found. My colleagues Andrew Turner, Warren Pearce and Sujatha Raman in particular know much more about this than I do)
Some history – evidence-based medicine
As is my want, I began to rummage around a bit in the history of the concept. As everybody knows, the concept of evidence-based policy has its roots in the concept of evidence-based medicine. To be more precise: “In the spring of 1990 the young McMasters University Internal Medicine residency coordinator, Dr. Gordon Guyatt, had just introduced a new concept he called ‘Scientific Medicine.’ The term described a novel method of teaching medicine at the bedside. It was built on groundwork laid by his mentor Dr. David Sackett, using critical appraisal techniques applicable to the bedside. However, the response from his fellow staff was anything but warm and inviting. The implication that current clinical decisions were less than scientific, although probably true, was nonetheless unacceptable to them. Guyatt then returned with a new title that described the core curriculum of the residency program: ‘Evidence-Based Medicine’ (EBM). The coined term appeared in a subsequent 1991 ACP Journal Club editorial.”
That was interesting, as it showed that ‘evidence-based’ medicine was indeed a ‘scientific’ concept, but that, like in the later case of ‘evidence-based’ policy the response from people who were introduced to that concept was “anything but warm and inviting”, although for different reasons. In the case of evidence-based medicine, those supposed to practice it resented the fact that their traditional ways of doing and knowing were deemed to be “less than scientific”, while in the case of evidence-based policy, those advising people on the policy-making process warned them that this might lead to policy-making based on nothing but science.
According to oral history, the concept of evidence-based medicine was often misunderstood. One founder pointed out that in contrast to what many believed: “it wasn’t old hat, impossible to practice, cookbook medicine, the creature of managers and purchasers, or concerned only with randomized trials.” Some of these misunderstandings seem to have carried over into current critiques of evidence-based policy-making.
Interestingly, again according to oral history, evidence-based medicine challenged ‘expert-based medicine’ and thus empowered young physicians, nurses and others to challenge authority and dogma. That’s something rather positive and it was picked up by policy makers.
Some history – evidence-based policy
Overall then, evidence-based medicine began to flourish in the 1990s. It seems that policy makers almost immediately wanted to emulate these efforts at making the practice of medicine ‘better’ in order to make the practice of policy ‘better’, that is to say, to engage in policy-making that can be questioned and improved, because it is based on evidence rather than ideology and dogma.
As Wikipedia points out: “Underlying many of the calls for ‘evidence based policy’ is often a (stated or unstated) concern with fidelity to scientific good practice, reflecting the belief that social goals are best served when scientific evidence is used rigorously and comprehensively to inform decisions, rather than in a piecemeal, manipulated, or cherry-picked manner.”
The term evidence-based policy seems to have first been used in 1996 by the statistician “Adrian Smith in his 1996 presidential address to the Royal Statistical Society, [in which] Smith questioned the current process of policy making and urged for a more ‘evidence-based approach’ commenting that it has ‘valuable lessons to offer’.” (Wikipedia) It was then taken up by the Labour party. Now it’s a phrase that has general, albeit increasingly contested, political currency.
Evidence-based policy – what is it, what should it be?
When looking at some introductory books on the matter of evidence-based policy, I found two which looked interesting – but there are hundreds I suppose. One, by Ray Pawson, is from 2006 and entitled Evidence-based Policy Making: A realist perspective, the other is by Justin Parkhurst entitled The Politics of Evidence: From evidence-based policy to the good governance of evidence. The first is worth a read and really quite fun. I’ll focus here on the second published in 2017, as it deals with more recent debates that inform the tweets I mentioned at the beginning. Here are a few salient points made in the book.
First, the books makes clear that given the “fundamentally contested nature of most public policy concerns, the use of evidence for policy has been described as ‘qualitatively different’ (Black 2001) than its use in technical decision-making arenas (such as clinical medicine)” (Parkhurst, p. 5).
Second, one should not only distinguish between the use of evidence in technical decision-making and the use of evidence in public policy decision-making, but also between the use of evidence in public policy-making and the ‘good governance of evidence’ in the public policy-making process.
Third, such governance of evidence-use in policy relies on spotting and dealing with two types of biases; what Parkhurst calls ‘technical bias’ and ‘issue bias’. According to Parkhurst, so-called evidence advocates are concerned that “political interests drive the misuse or manipulation of evidence” – leading to technical bias -, while critical policy scholars highlight that “appeals to ‘evidence-based policy’ can depoliticise political debates” (see Parkhurst, p. 6) – leading to issue bias. “The first form of bias broadly reflects the value of scientific fidelity, while the second broadly reflects the value of democratic representation.”
The governance of evidence in public policy-making
What’s the way forward – how can evidence-based policy be governed in such a way that these biases are avoided or, indeed, these values reconciled?
Parkhurst has a proposal: “efforts to improve evidence use will ultimately require building systems that work to embed key normative principles about evidence utilisation into policy processes – systems that can be said to govern the use of evidence within policymaking. Therefore, in order to move the EBP field forward, it is necessary to consider how to establish evidence advisory systems that promote the good governance of evidence – working to ensure that rigorous, systematic and technically valid pieces of evidence are used within decision-making processes that are inclusive of, representative of and accountable to the multiple social interests of the population served.” (p. 8)
To this I would add: the good governance of evidence project should extend its reach from the use of evidence in policy-making to the collection of evidence in science, i.e to ‘the good governance of science’. At the moment the governance of science is in crisis, as a regime of perverse incentives is rolled out, such as REF and impact, and universities and other public agencies are becoming commodified, industrialised, marketised and financialised. This compromises the gathering and value of scientific evidence. Another danger is posed by the entanglement of science and politics fostered by these developments.
Evidence is a tool, a technology; it can be used for good or ill. It has to be collected and used with care. This includes creating the right conditions for scientists to do their job.
I think it is important to make a distinction between (scientific) evidence and the use of (scientific) evidence in policy and politics – which can lead to the political use of that evidence. In my view, various types of scientific evidence (from quantitative modelling to qualitative case-studies) are a necessary but not sufficient conditions for devising policies that deal with health, medicine, technology, innovation, climate change (mitigation, adaptation) and so on. Scientific evidence is a key component of modern policy analysis and policy decisions, but it is, of course, always used alongside and traded against other components, such as political and social values.
Scientific evidence gathered and used by policy makers is always partial, limited and contested; it can come from reliable sources or unreliable ones; it can come from a narrow range or plurality of sources or a broad one; it can be used and misused in the pursuit of power; it can be dismissed or it can be given unwarranted priority. There can be “cherry-picking, obfuscation or manipulation” (Parkhurst, 2017: 4) All these things can happen. What always does happen however is that scientific evidence is interpreted by policy makers in light of pre-existing knowledge and pre-judices. After that policy makers have to use their power of judgement and their political power to come to a decision – their decision.
But I don’t see how policies that affect how we live, work and die in modern high-tech societies can do without scientific evidence or input, i.e. can NOT be evidence-based or, to use a weaker form of words, can NOT be evidence-informed. The political processes of evidence-gathering and evidence-use (including the gathering and use of scientific evidence) should, of course, be governed by shared normative principles, as outlined by Parkhurst. They should also be informed by the ideal of evidence-based policy.
And finally, scientific evidence ALONE, indeed any type of evidence alone (see Parkhurst, p. 5) will never sway policy makers and neither should it. This seems to be quite a preposterous proposition to me. We should certainly not dismiss the notion of evidence-based policy based on the assumption that this has ever been proposed. Has it? I’d love to know.
PS. I had just finished writing this when I saw this tweet which is a direct counterpoint to what I have said here. Discuss!
Image: Wikimedia commons
Kathryn and I attempt to summarise the literature vis-a-vis EBM and EBP here https://www.nature.com/articles/s41599-017-0045-9
Governance, as you point out, is fundamental. What counts as evidence, and at what time, is a vexed question. Critics of EBP might suspect that definitions of “evidence” are often very narrowly drawn; for example, are we moving towards policy-making that relies too heavily on randomised trials?
So definitions are key, like other well-wron phrases such as sustainable development, one has to be sure what one is talking about, lest we have an excess of interpretive flexibility (i.e. too vague).
I wrote an article partly supported by Making Science Public about this in relation to local climate policy http://www.ingentaconnect.com/content/tpp/ep/2014/00000010/00000002/art00003 (open access)
Thanks! Just the sort of input needed. The Parkhurst book references your work quite a bit 🙂 I had, I have to confess, forgotten about the three lessons article 🙁
I had also forgotten about your blog post from 2014!! https://blogs.nottingham.ac.uk/makingsciencepublic/2014/07/16/evidence-based-policy-data-has-its-limits/
Isn’t there an irony to objections to idea of evidence-based policy? If Silvio Funtowitz has some objection to the concept of using information (evidence) in the decision making process because it’s somehow only accessible to some elite, than surely the same can be said of any information that is derived from any kind of process of investigation. Consequently, any experts associated with that process can be regarded as priviledged and supporting some kind of PR wing of their enterprise. For example, one could then dismiss any STS derived information (evidence) because communicating it is simply the PR wing of their enterprise, and the purveyors of that information are somehow priviledged/elitists (or whatever it is that Silvio Funtowitz regards as the position of those who have priviledged access to this information/evidence).
Okay, maybe I’ve misunderstood the fundamentals of the objection, so would be happy if it were explained more clearly.
That is a good question and I think one that STS people need to ask themselves.
To add another twist to this discussion:
If you actually read scholarly literature in Public Relations, or listen in to the conversations in the professional communities (e.g., PRSA), you’ll find that PR professionals take responsibility for critiquing the organizations they represent. Basically, an organization cannot remain on mutually respectful terms with its stakeholders when it is acting unethically. So an organization’s PR professionals act as ethics watchdogs internally, to make their job of maintaining external relations easier–or even possible.
In this sense, STS is indeed the PR arm of science. 😉
That is interesting. But does STS really see itself as an ethics watchdog? Isn’t the post-ELSI agenda to genuinely collaborate with scientists on equal terms? For that to happen we need to develop mutual understanding. Wielding the PR word doesn’t contribute to that process, I suspect, as for actual (perhaps too idealistic) science communicators PR is a big no no.
The problem is much worse than just irony. Without science who is going to put a limit on the knowledge claims of the elites? What is the alternative to evidence-based policy? Elite talking-point based policy? Unconstrained trampling of the population by crony capitalists?
On my one non-strike day, I will not be engaging at length here. However, I think dismissing a field of study based on second-hand reporting of tweets can probably be filed under “sub-optimal”.
Without wishing to speak for Clark, I suspect the point is that science communciation is extremely well-funded by industry in many areas (eg GMOs), and such communication has certain values inherent to it. So science communication is not PR per se, but things that are classed as “just the facts” such as GMO discussions are extremely hard to extricate from the values that are driving the communciation in the first place.
I am not dismissing it. I am submitting it to critique. That is standard practice in the social sciences, isn’t it? And questioning a few tweets doesn’t mean questioning a whole field, even though the tweets seem to reflect quite wide-spread attitudes and opinions, as far as I can make out.
My point is that the question is built on a straw man characterisation (also standard practice in many sciences!) No-one is saying that any experts could be dismissed just because of their perceived interests; that is an absurd relativist position, which fundamentalists made a lot of hay out of in the science wars but I would hope we would be beyond characterising as a mainstream social science (STS) position.
The danger with “evidence-based policy” is the embedded claim that the evidence speaks for itself in some magical way, which it never does. it requires interpretation which is in part based on our values. If you want to take the current example of the strike, the evidence on which USS’s decision is disputed. It is partly disputed because the process has been so opaque, and partly because the suspicion is that UUK’s position is being driven by a set of values that are biased against collectivist provision. They are entangled. The way past this is not to throw one’s hands up in the air and go “it’s all relative”, but to find better, more collective ways to govern the production and use of evidence. Strides have been made in this direction in some areas of health (mental health, autism, HIV to name three disparate examples).
Sorry, I actually replied to something addressed at ATTP, I think. But be this as it may. The question about who shapes what Parkhurst calls the good governance of evidence remains – so which experts are allowed in and which aren’t? Do scientists trump STS experts or vice versa? This is a really difficult issue for governance. I am no expert on evidence-based policy but do people really still think, or did they ever think that evidence speaks for itself? Isn’t that also a strawman? Similar to the claim that scientific evidence alone does the trick so to speak? And yes the USS strike is probably a good case-study as to what evidence is admitted and why and that critiquing those wielding power, the UUK, is necessary. Interestingly, this critique has, by some, been framed as ‘evidence-based’, e.g. Michael Otsuka: “What follows is an evidence-based correction to @UniversitiesUK’s (even by its own standards) shameless misrepresentation of the costings of @ucu’s proposal. First UUK’s shameless distortion from yesterday’s press release. UUK wrote: 1/….”
I couldn’t tell myself. My comment was actually a question and was essentially trying to suggest that noone has some kind of priviledged position, be they scientists, or those who study science and society. Scientists should be careful as to how they engage in the process of evidence-based policy making, but the same should be true of others who feel that they have relevant expertise. And, as it sometimes seems (even if it isn’t the intent), if one can undermine scientific evidence for various reasons, then the same could apply to any kind of evidence, including that being used to undermine the scientific evidence (I was trying to highlight an irony, which maybe isn’t an irony, but it might be nice if people at least appeared to give this some thought).
I also have never encountered anyone who seems to think that evidence speaks for itself in some magical way (this would seem a rather bizarre thing to think). I have encountered many who have views based on the evidence, but that seems an entirely reasonable thing to do in a democracy. They may also express dismay at how they see policy makers using evidence in some cases. Again, seems reasonable in a democracy. There may even be some who strongly highlight the available evidence when commenting on policy, but – again – this doesn’t necessarily mean that they think it speaks magically, they just happen to have strong views as to what should be done, given that evidence. Again, democracy.
Yes, it is terrible to sub-optimally dismiss entire fields of study.
In the interests of maybe trying to achieve something, the above seems obviously true and I doubt there are many scientists who would disagree. If this is essentially what people mean when they critique the role of scientific evidence in the policy process, why can’t they just say this?
Sidestepping the issue of formulating policy based on scientific evidence for a moment, I have proposed a grand bargain in which parties will agree to pass legislation on the condition that any new law will automatically sunset after a specific period if independent, scientific research does not show that it has accomplished what it was designed to do, based on metrics agreed to at the time of passage. Essentially, I am proposing nothing more than the testing of a hypothesis. By agreeing on goals and metrics ahead of time, such a bargain would leave little room for non-scientific interpretation of the effectiveness of the legislation.
People on both sides of the aisle here in the United States tend to believe their opponents’ policy proposals are nothing more than demagogic talking points, designed to secure electoral advantage rather than to solve a problem. A bargain might be struck if the proponents of a measure are willing to have it automatically phase out (without a future vote) if it does not accomplish the stated goal in accordance with metrics set up at the time of passage.
If this kind of evidence-based analysis were to become the norm, the effects would be salutary. Those proposing policy would be much more likely to formulate it based on scientific evidence, knowing that if it does not work, it will judged as a failure and automatically discontinued—something that does not happen with purely ideological proposals. Such a system would focus legislators on solving problems, rather than on political grandstanding.
Here’s a functionally similar proposal that’s already being used:
Of course, it uses market as opposed to deliberative mechanisms, but the outcome is similar: adaptive management of societal problems.
That is certainly a very interesting proposal! One that I haven’t come across before, but really worth thinking about. Having to think ahead about what would make a policy successful and how such a success could be evidenced (to use a stupid word) would, may be, force policy makers to change how they think about evidence in the policy-process itself. It would make it less easy to dismiss evidence that doesn’t suit ideological ends or manipulate it for ideological ends or indeed make policies just on a whim.