December 12, 2012, by Warren Pearce
Inside climate science: the opening and closing of IPCC expertise
This is a guest post by the University of Nottingham’s Paul Matthews – outlining what he can (and can’t!) divulge about the IPCC’s peer review process.
The Intergovernmental Panel on Climate Change (IPCC) is the scientific body established by the United Nations to provide assessments of current knowledge in this complex and controversial field of study. The IPCC is divided into three sections or Working Groups. WG1 deals with the basic physical science, WG2 looks at impacts and WG3 considers possible mitigation strategies. Each WG follows a slightly different timetable (with WG1 being ahead of the other two) but adopts the same basic procedure. The IPCC authors write a first draft (FOD) that is subjected to expert review. A second draft (SOD) is then produced, in the light of these review comments. The SOD is reviewed by experts and government representatives, and then the final version of the Report is prepared and published. So it’s a bit like writing a paper for a journal – except that you have (for WG1) over 200 authors, 600 reviewers and 20,000 individual comments! For WG1, the review of the SOD has just been completed, the deadline for submitted comments being the end of November.
How to become an IPCC reviewer
So how does the review process actually work? Who is involved? Who counts as an ‘expert’ reviewer? I was discussing these questions with a climate scientist who said that anyone could register as a reviewer. Intrigued and somewhat sceptical about this, I went to the IPCC reviewer registration site (now gone, so no link). This asked for relevant publications (I put in a few, only rather indirectly relevant), a comment on relevant expertise, and finally asked you to click a button that said basically ‘yes I am an expert’. I was pleasantly surprised to get the response:
Registration AS EXPERT REVIEWER Confirmed
Your confirmation number is: 948
followed by a formal letter of invitation to take part in the review. Sceptical blogger Anthony Watts did the same thing and was shocked to be accepted as IPCC reviewer 1029. The invitation letter asks for comments to be presented constructively and with clear specific suggestions, and emphasises in bold that the draft reports may not be cited, quoted or distributed. You are provided with a password to download the draft reports from the IPCC website, but before you can do this you have to click a box saying that you agree to abide by the no-distribution rule. The letter also explains that after the report is published, all the reviewer comments and the author responses to these comments will be made available, and these comments will not be anonymous. (for the previous IPCC report these can be found here – browsing through these files gives some indication of the vast amount of detailed work involved).
Questions and controversies
Overall, my impressions of the process were quite positive. It was gratifying to see, when reading the SOD, that quite a few – about half – of the comments I made on the FOD had been acted on. It’s instructive to see how the process works and as a partial insider you appreciate the difficulties involved.
But there are a few questions raised, relevant to the ‘Making Science Public’ theme. Foremost among these is the confidentiality issue. It’s quite frustrating that this document detailing the current state of knowledge about climate science has been written and is in pretty good shape, but may not be quoted and won’t be made public for a year or so. For example, I know what the report says about hurricanes, and it would be interesting to discuss this in the light of Warren’s recent talk and blog post on Sandy – but I’m not allowed to! Stephen McIntyre of the Climate Audit blog, who was a reviewer of AR4, says that he did register as a reviewer for AR5 but did not download the report to review it because he was not prepared to agree to the “Do not cite, quote or distribute” rule. He also says that these confidentiality procedures have been tightened up, despite the fact that an Inter-Academy Report had emphasised the importance of transparency. But of course there could be problems if the whole review process were completely open. Bloggers could organise campaigns to coordinate dozens of critical comments on the same issue.
A second minor irritation is that when reviewing the SOD, you don’t get to see the explicit response of the authors to your comments on the FOD, even though we know these will eventually be released. You have to infer the extent to which your comments have been accepted by reading the text of the SOD, which can be tricky if the text has had a major reorganisation.
Another question is whether review comments should be anonymous, as is normal in academia. The IPCC considered this at a meeting last year, but decided against it. The arguments in favour are that any bias or prejudice against a particular reviewer is avoided, or to put it the other way around, the IPCC could not be accused of any such bias. On the other hand, associating each comment with the name of the reviewer may help to ensure that comments are made in a respectful and professional tone.
[Image credit: Ajar, by Romeo66]