December 12, 2012, by Warren Pearce
Inside climate science: the opening and closing of IPCC expertise
This is a guest post by the University of Nottingham’s Paul Matthews – outlining what he can (and can’t!) divulge about the IPCC’s peer review process.
The Intergovernmental Panel on Climate Change (IPCC) is the scientific body established by the United Nations to provide assessments of current knowledge in this complex and controversial field of study. The IPCC is divided into three sections or Working Groups. WG1 deals with the basic physical science, WG2 looks at impacts and WG3 considers possible mitigation strategies. Each WG follows a slightly different timetable (with WG1 being ahead of the other two) but adopts the same basic procedure. The IPCC authors write a first draft (FOD) that is subjected to expert review. A second draft (SOD) is then produced, in the light of these review comments. The SOD is reviewed by experts and government representatives, and then the final version of the Report is prepared and published. So it’s a bit like writing a paper for a journal – except that you have (for WG1) over 200 authors, 600 reviewers and 20,000 individual comments! For WG1, the review of the SOD has just been completed, the deadline for submitted comments being the end of November.
How to become an IPCC reviewer
So how does the review process actually work? Who is involved? Who counts as an ‘expert’ reviewer? I was discussing these questions with a climate scientist who said that anyone could register as a reviewer. Intrigued and somewhat sceptical about this, I went to the IPCC reviewer registration site (now gone, so no link). This asked for relevant publications (I put in a few, only rather indirectly relevant), a comment on relevant expertise, and finally asked you to click a button that said basically ‘yes I am an expert’. I was pleasantly surprised to get the response:
Registration AS EXPERT REVIEWER Confirmed
Your confirmation number is: 948
followed by a formal letter of invitation to take part in the review. Sceptical blogger Anthony Watts did the same thing and was shocked to be accepted as IPCC reviewer 1029. The invitation letter asks for comments to be presented constructively and with clear specific suggestions, and emphasises in bold that the draft reports may not be cited, quoted or distributed. You are provided with a password to download the draft reports from the IPCC website, but before you can do this you have to click a box saying that you agree to abide by the no-distribution rule. The letter also explains that after the report is published, all the reviewer comments and the author responses to these comments will be made available, and these comments will not be anonymous. (for the previous IPCC report these can be found here – browsing through these files gives some indication of the vast amount of detailed work involved).
Questions and controversies
Overall, my impressions of the process were quite positive. It was gratifying to see, when reading the SOD, that quite a few – about half – of the comments I made on the FOD had been acted on. It’s instructive to see how the process works and as a partial insider you appreciate the difficulties involved.
But there are a few questions raised, relevant to the ‘Making Science Public’ theme. Foremost among these is the confidentiality issue. It’s quite frustrating that this document detailing the current state of knowledge about climate science has been written and is in pretty good shape, but may not be quoted and won’t be made public for a year or so. For example, I know what the report says about hurricanes, and it would be interesting to discuss this in the light of Warren’s recent talk and blog post on Sandy – but I’m not allowed to! Stephen McIntyre of the Climate Audit blog, who was a reviewer of AR4, says that he did register as a reviewer for AR5 but did not download the report to review it because he was not prepared to agree to the “Do not cite, quote or distribute” rule. He also says that these confidentiality procedures have been tightened up, despite the fact that an Inter-Academy Report had emphasised the importance of transparency. But of course there could be problems if the whole review process were completely open. Bloggers could organise campaigns to coordinate dozens of critical comments on the same issue.
A second minor irritation is that when reviewing the SOD, you don’t get to see the explicit response of the authors to your comments on the FOD, even though we know these will eventually be released. You have to infer the extent to which your comments have been accepted by reading the text of the SOD, which can be tricky if the text has had a major reorganisation.
Another question is whether review comments should be anonymous, as is normal in academia. The IPCC considered this at a meeting last year, but decided against it. The arguments in favour are that any bias or prejudice against a particular reviewer is avoided, or to put it the other way around, the IPCC could not be accused of any such bias. On the other hand, associating each comment with the name of the reviewer may help to ensure that comments are made in a respectful and professional tone.
[Image credit: Ajar, by Romeo66]
Thanks Paul, really interesting post. Have you got a view on how useful these huge reports are for either the advancement of scientific knowledge, or expert advice to policymakers? IPCC reports are at the other end of the spectrum from the ‘extended peer review’ being carried out across a range of climate blogs. These are obviously very nimble and very open, but with that comes a big spread in the nature of the comments. IPCC reports seem huge and lumbering in comparison, but does that mean they are no longer useful, are ‘out of date’ before they are published etc? Or does the journal article remain the gold standard, striking a happy medium between these two extremes?
That question might be better answered by a real climate scientist. I think the aim of the reports is to provide a summary of the extent of current knowledge (rather than actually advance knowledge), and as such they provide a useful starting point for future research – for example a paper may introduce a topic by saying something like ‘it is generally agreed that…’ or ‘one of the major areas of uncertainty in our current knowledge of climate science is …’ and quote the IPCC report as an authoritative reference. But yes I think most scientists would say that the journal article is still the gold standard, though that may be changing, as discussed in your last post.
Science-perspective: If I reference an IPCC-chapter, it is usually the lazy way of getting a “reviewed” reference and a lazy way of getting the “status quo” of our knowledge. As such, they are nice and also in a way important especially for stuff which otherwise would need a large number of individual references. The reports also serve well as a starting point although they are to some extent out of date when they are published. They nevertheless provide potentially a good review of the state of the science. And I would say, reviews always are in danger to be out of date.
With respect to expert advice I guess (!) the same holds. That is, they are easy references which have to be supplemented by more specific assessments and specific studies in the “problem” at hand. I am not familiar with policy making, but I would guess that also there the reports (or at least their shorter summaries for policy makers) are the primary reference. I think of the regional agency-reports I looked at most at one point or another quote the IPCC-ARs so they don’t have to review all the basics.
Nevertheless, I think there are good arguments – especially from the WG1 perspective – to abandon the full-blown ARs and only present more specific assessments.
Thanks Oliver. Good point re what referencing IPCC symbolises, have done this myself in the past. Guess they serve a similar function to Cochrane meta-reviews in medicine – wonder how they differ in methods? If they are also key in policy-making, why do away with AR1? Because the ground in there is now well-established? Could specific assessments be completed more quickly, or would they just be completed on a staggered rotation basis?
If I understand AR5 correctly, it’s anyway only thought as an update to AR4. So specific reports would allow 1) to focus on those topics with changing “knowledge” or with rapid advances and 2) would allow to generally update the knowledge more quickly. e.g. if the paleo-community is faster than the decadal-prediction community, the former doesn’t need to wait for the latter. The same applies, if WG2 thinks it needs an update although WG1 sees no need to update now. (or at least in that direction).
Interesting news this morning: the entire Second order Draft (SOD) of the WG1 report has been leaked. Given that that over 1000 people signed up as reviewers, it could be argued that it’s surprising that it took so long for someone to leak it. The timing of the leak is significant, coming shortly after the deadline for the reviewer comments. Thus the only argument I gave in the post for not making the drafts public (the possibility of coordinated campaigns of reviewer comments) no longer applies.
Perhaps you need to change the picture at the top of the post Warren – the door is now wide open!
Thanks Paul. Yes, perhaps I should change the image – a flapping barn door perhaps?
This raises some really key questions. Technological change has facilitated ‘information wants to be free’, and increasingly we see institutions having to deal with greater calls for transparency. As regards the ‘finished product’ in research – the journal article – the open access movement is having significant effects in this area.
But does/should this apply to the creation of knowledge? Gavin Schmidt claims that authors need early drafts need ‘safe space’ away from ‘politicised cage match’. Is this really the case? Is Schmidt right that in the early stages of what we might call the ‘production of knowledge’, scientists benefit from having a ‘safe space’ in which to deliberate? Or is this just a hangover from a previous age, and that we all need to get used to a new age of open deliberation? Mark Carrigan’s work on continuous publishing points a way to how this might work in social sciences, turning much more of one’s internal thoughts out into public. Does the IPCC leak show that attempts at secrecy are outdated, and that academics need to adjust their practices to the new technological realities (no matter how painful the transition)? If technology can’t be stopped, what are the implications for the academic ‘end product’?
Confidentiality is not a big deal. When you review a paper for a learned journal, you also promise to keep mum and not to knick their ideas. I reviewed two really stupid papers this week, but I won’t tell you who wrote them, and I reviewed one paper with brilliant data and appalling analysis — it itches, but I wont’ can’t shan’t do it better.
The problem with the IPCC mode of review is that drafts were bound to be leaked, and that those who leak have a different agenda than the IPCC. You can never control a message, but early spin is the spin that sticks.
Thanks Richard. So does this mean that the IPCC needs to ditch any hope of confidentiality in the future, as it can’t hope to scale up the concept from journal peer review, or that the whole IPCC AR model is out of time?
I think the IPCC should put early drafts on the web for open review.
I also think that Lead Author Meetings should be live webcast, if only to put all conspiracy theorists to sleep.
Boredom could be an advanced form of mind control…
I am unsure. I understand the wish that early drafts should be confidential. We usually do not write our papers openly. I also understand that the lead author’s shy away from being publicly accountable for every statement they make, for every formulation they choose in the early stages of the process. However, it certainly was unnecessary, if not even unwise, to “hide” the final first order and second order drafts. Making them publicly available wouldn’t have compromised the process. Maybe Keeping them behind “walls” follows the logic of big-publishers. The publicity is bigger if the public doesn’t know the content from early on?
In a sense the process was open (anybody could review) and closed at the same time. So indeed the drafts were bound to leak. And since parts of the FODs were leaked earlier as well, one should have changed the procedures then. Drafts open but still formalised review-system. However, that’s spilled milk. And the publicity is going to be large enough when AR5 is published officially.
Good points re the process being open and closed. As becoming a reviewer was so straight forward, perhaps most of the people who were interested in reading an early draft had already read it! The PR aspect is also relevant – no doubt the ‘big reveal’ is seen as effective. Although interesting that doesn’t seem to have been that much coverage of the leak in mainstream media. My suspicion is that it’s because interest in climate change as a whole is down. WIll see what happens when AR5 is out proper.
The IPCC is politicized by its very nature. It is not a scientific body but a hybrid. As Matthew explains above, government representatives have a big role in the drafting.
The early spin that sticks: Andy Revkin has drawn attention to the changing nature of who tries to sting first. It used to be people from the opposite side of the debate.