October 23, 2015, by Warren Pearce
Methodological clarity required when publishing social science in natural science journals
This is a joint post with Greg Hollin.
The latest issue of Nature Climate Change features a Correspondence from Peter Jacobs and colleagues which concerns a recent Letter that appeared in the same journal; our Reply is also published. We do not wish to deny that there are real and significant differences between ourselves and Jacobs et al but we think the correspondence also says something about the publication of qualitative research studies in prestigious natural scientific outlets. This Correspondence/Reply received three-peer reviews: one appeared unsure if our reply was worthy of publication; one felt Jacobs et al should be ignored; and a third, rather presciently, felt that both social scientists and natural scientists would think they had ‘won’ and return to their houses none-the-wiser. Given that Nature Climate Change has published many calls for qualitative and/or interpretive studies from the social sciences and humanities, we’d like to muse, very briefly, on what might be learned from this exchange for those writing, or reading, reports in the future. From our perspective, we’d like to highlight the importance, and difficulty, of inductive research.
We could think of hypothetic-deductive, hypothesis driven, research as being “top-down” – one starts with a theory or a question and then interrogates a data set in order to see how that theory stands in relation to the data. Inductive research is quite different and is “bottom up” – you start with the data, see patterns or interesting things, and the theories and broader claims are integrated later. This latter approach is certainly the one which we took in our Letter; the two main theoretical sources in that piece – Jeanne Fahnestock’s work on meaning in science (£) and Susan Leigh Star’s writings on certainty in science – were only incorporated after we had analysed the press conference transcript itself. This approach is incredibly common in the social sciences; Grounded Theory, the archetypal inductive approach, is quite possibly the most used research method. Inductive research is, however, far less common in the natural sciences (although some claim that Big Data is changing this).
When reading back the methods section of our Letter is noticeable that the word ‘inductive’ does not appear at all and the role of theory in our coding scheme (i.e. it played a largely insignificant role) isn’t as well explained as it might be. To a social scientific audience we don’t think this would be problematic but in this context we think it proved to be so; there is the strong sense that Jacobs et al think that we didn’t test our hypotheses rigorously but this is precisely because there weren’t any hypotheses. A perfect example of different answers arising from an unacknowledged difference in questions rather than anything fundamental about answers.
As calls (and pushes!) for interdisciplinary research intensify, our experience at the sharp end within a high-profile journal have taught us two basic but important lessons for future engagements: i) qualitative researchers need to be clear about what their methods are when they’re using when publishing in natural science journals, and ii) those more accustomed to hypothetic-deductive approaches need to at least be aware that qualitative methods frequently use different rules to those they most commonly encounter.
[…] Pearce and Greg Hollin have just written a blog post about their experience in writing their article and the response to our comment. What they seem to […]
In your response to Jacobs et al. you repeat your original misrepresentation about Jarraud’s response to a journalist’s question; i.e. you say:
“If asking about the decade-long pause is an “ill-posed scientific question”, as asserted by Michel Jarraud during the press conference….”
But it’s clear from listening to the exchange (and reading the transcript) that Jarraud wasn’t asserting that “asking about the pause” is an ill-posed question. The “ill-posed” question was “…how much longer will the so-called pause or hiatus have to continue before you would begin to reflect that there is something fundamentally wrong with the models?” And that specific question is ill-posed since it doesn’t have a directly sensible answer (the most appropriate answer might be “it depends…”).
I find it incomprehensible that so much of your argumentation is based on misrepresentation of what scientists say. Michel Jarraud simply did not “assert” what you asserted. Why can you not let people’s (scientist’s) words mean what they say?
Isn’t your inductive/deductive dichotomy a red herring? Much of physical scientific research is an intertwined mix of induction and deduction. In many cases measurements and analyses are made with the expectation that the observations will be valuable in hypothesis generation (inductive), rather than in hypothesis testing (deductive); in some cases there may be a mix of expectations. Much of structural biology and quite a bit of molecular biology, molecular medicine, drug development etc. is like this, and the development of novel approaches and technologies (which play such an important part of scientific advance) often have an inductive underpinning (“we’ve found a way of collecting some really nice data on such-and-such – let’s do this and see what it might tell us”). Long term collection of surface or atmospheric temperatures, collection of ice-cores for analyses of trapped gases etc. were not done to test a hypothesis – they were done with the expectation that the measurements would inform understanding, would stimulate the development of new theories, and might allow us to choose between different theories.
Far, far more important than inductive/deductive considerations (which scientists give rather little thought to in my experience) is an ability to make sound measurements and logical analyses and interpretations, and above all an honest relationship with the evidence…
Hi Greg and Warren,
Interesting reflections on induction vs deduction, but to my mind largely irrelevant in explaining why we differ in opinion regarding the IPCC press conference.
This difference in opinion boils down to you misinterpreting statements relating to timescales and basing the whole remainder of your argument on a false premise.
The “warmest decade” comments were clearly made in the context of the long-term trend. I’m struggling to understand your position on this aspect. Do you fail to see this context or do you consider it irrelevant, or am I missing something else?
The so-called “pause” or slowdown wasn’t dismissed.
Whether these mistaken interpretations were arrived at via induction or deduction is perhaps interesting, but doesn’t explain why we see this differently.
See more at https://ourchangingclimate.wordpress.com/2015/10/24/from-complex-clarity-to-nuanced-misunderstanding-response-to-hollin-and-pearce/.
I too have written a post about this. For the lurkers, I should probably ackonwledge being one of the authors of the response to your original article.
https://andthentheresphysics.wordpress.com/2015/10/23/clarity-of-meaning/
I was going to make similar points to those made by Chris and Bart. I don’t think that science is typically deductive rather than inductive. I think it is very mixed, with it sometimes being deductive, and at other times inductive. Also, as Bart points out, it’s not clear why it is really relevant here. However one approaches a piece of research, the result should be a reasonable representation of what the data suggests, irrespective of whether it was an inductive, or deductive, piece of research.
Sou also has a blog post at …
http://blog.hotwhopper.com/2015/10/the-ipcc-climate-message-is-clear-based.html?showComment=1445680865950#c6980065020972015357
I have made several rather critical comments at both Sou;s and ATTP’s.
At this point I only really have one burning question.
What is the original source for your statement in your previous blog post …
“Drawing on meaningful information like ‘the hottest decade’ proved problematic for the scientists for it is hard to see why the short-term increase in temperature during ‘hottest decade’ is very different from the short-term decrease in temperature witnessed during the 15-year ‘pause’.”
Specifically the “short-term decrease in temperature witnessed during the 15-year ‘pause’.” part?
A direct link to this underlying claim would greatly be appreciated.
Thank you.
Hi Everett,
I’ve just looked again at the blog post, thanks for drawing my attention to it. We’ve used looser language than we should have done there. It would be much clearer if it said at the end of the sentence ‘prima facie’. So we are *not* saying that the two phenomena are the same, based on scientific knowledge. What we were saying is that beyond an expert climate science context, they appeared to be similar at first blush. This has become much clearer with the literature that has been published since the press conference, it was less obvious at the time the press conference was held. As cited in our Supplementary Information for the Reply, the Cowtan and Way paper published post-AR5 makes a very plausible case that gaps in surface temperature coverage may have a lot to do with the apparent pause.
In our recent paper on climate change communication http://onlinelibrary.wiley.com/doi/10.1002/wcc.366/abstract (sorry £ but email me if you are interested in finding out more), we make a plea for a more reflexive public understanding of how climate science works and what it can and cannot tell us. For example, temperature data is never ‘straightforward’, as some critics like to argue it was in the good old days. A better understanding (and explanation) of what goes into the making of scientific knowledge. This is one of the reasons we cited the Chasing Ice film in our Reply as a good example of making climate change meaningful, as it provides a warts and all account of collecting some very powerful and persuasive data.
Hope this helps.
I’m a bit lazy, and don’t really want to take the time to read the entire backstory, but I’m hoping you’ll answer a question. You say:
=>> “…Inductive research is quite different and is “bottom up” – you start with the data, see patterns or interesting things, and the theories and broader claims are integrated later. This latter approach is certainly the one which we took in our Letter;…”
What data are you referring to? Can I assume that you mean data that show that people were confused about the certainty of evidence about climate change – as a result of the press conference in question?
If so, what evidence did you have to quantify those data and you conclusion about those data (the cause of “confusion), and in particular, what is the evidence of your control for potentially conflating influences such as “motivated reasoning” on the part of those who say they were confused? As I recall, the main component of you “data” was Rose saying that he found the IPCC statements to be contradictory (with an implication that the IPCC was selling the public a hill of goods). How did you determine that Rose (or others) weren’t more likely confirming their biases about the political motivations of the IPCC?
Warren –
While I’m waiting to see if my comment exits moderation…
Here’s the thing…I kind of agree with you in the sense that I think that arguing about the technical merits of whether there’s been a “pause,” or whether referring to a comparison between decades is the same as using the “pause” to conclude that there are basic flaws in the “mainstream” argument about climate change, are too abstract to change public opinion.
I think that in a sense, you’re right in that to the extent that there are those in “the climate science community” who think that “scientific knowledge alone” will determine how the general public views climate change, then those scientists are probably mistaken.
But all that given, my sense is that what you’re mostly doing is exploiting those truths to pursue an agenda. My sense is that underneath your thesis lies a view that “the climate science community” seeks to use fear-mongering to persuade the public because pure science won’t – and in that sense, the existence of “skepticism” can be attributed to poor communication practices from “the climate science community.” Thus, your agenda is to assign causality for “skepticism” at the feet of “the climate science community.”
Such a view assumes that “skepticism” towards climate change is rooted in what “the climate science community” does or doesn’t say. Such a view basically basically ignores the ubiquitous evidence of an association between “skepticism” about climate change and political ideology – an association that strongly suggests that people of a certain ideological orientation basically go looking for reasons to be “skeptical” about climate change irrespective of the specifics of what “the climate science community” says. In other words, “skeptics” start out with judging “expert” opinion on the basis of whether the “expert” is aligned with them ideologically. Then they filter that evidence in a process of reverse engineering to confirm their starting biases.
But if I’m wrong, (and I’m right about what you meant by “data” in your earlier statement), then it seems to me that you should, indeed, be able to provide the “data” that you describe which shows the causality that lies underneath your thesis about why people were confused by the press conference in question.
Hi Joshua,
Apologies for you being stuck in moderation. I don’t work at weekends, so posting this on Friday afternoon was sub-optimal. Thanks for these comments (and previous ones elsewhere), which engage with some key themes that we didn’t have space to draw out in the paper. I’ll try to respond succinctly.
The data we refer to here is the transcript of the press conference.
Of course, this is a reasonable point re Kahan etc. However, the point of the paper is *not* to demonstrate whether or not certain journalists demonstrated motivated reasoning. Most, if not all, arrived at the press conference with particular views on climate change. As we mildly (probably too mildly) say in the original paper: “We do not wish to suggest that David Rose was particularly sympathetic to the IPCC prior to the press conference” and linking to two fairly polemical articles (neither of which I agree with btw). We said this explicitly as a qualitative qualification of our argument. Perhaps one could find a way of controlling for motivated reasoning, and maybe we could explore that in a subsequent paper. However, the fact of the matter is that journalists do not play by the same rules as scientists. So a part of making climate change meaningful requires being attuned to these biases, as I would be very sceptical that one would be able to erase them.
Great, thanks! This is borne out by Jacobs et al’s Correspondence imho.
I’m sorry to read this, as this is definitely *not* my intention. I’ve never accused climate scientists of fear-mongering. I certainly wouldn’t blame the existence of “scepticism” on poor comms (although some would). The literalism/lucidity conundrum doesn’t have any universally satisfactory answers see https://blogs.nottingham.ac.uk/makingsciencepublic/2012/11/14/short-circuiting-the-language-of-sandy-how-to-balance-literalism-and-lucidity/ .
I *would* argue that attempts to depoliticise the climate policy debate (notably through the spurious fact/value distinction that the IPCC has been saddled with since birth) have laid the foundations for debates which have caused a lot of people to become stuck in intellectual mud over whether or not there has been a pause, what the exact value of climate sensitivity is, and quantifying the level of climate science consensus. The cultural and political questions about what we do with our climate knowledge dwarf these scientific curios imho. At the end of our original paper we refer to this, but this has been overlooked in favour of more chat about the pause.
How? You seem to have even said that in your response to Jacobs et al. I fail to see how, given that the response suggested (unless I missed it while proof-reading) no such thing.
Hi all,
Just to clarify the purpose of this blog – the purpose is not to address substantive issues pertaining to the paper itself. The questions being raised here by Chris and Bart are addressed in the paper and/or the reply to it, including the supplementary information. One point of clarity may be worthwhile, however: our data entirely relate to press conference and were analysed qualitatively; our piece is primarily concerned with the press conference and neither with physical science nor any journalistic activity which took part outside of it.
In this light, the inductive/deductive discussion is a forward looking exercise (and it is explicitly described as such) which relates to any future publications appearing in outlets like Nature Climate Change. We don’t pretend that what is presented is anything other than an incredibly simplistic view but it is also true that, both privately and publically, we have fielded a significant number of questions about what measurements we used, what our null and/or experimental hypotheses were and, indeed, what is meant by inductive research. Such statements seem to demonstrate a lack of familiarity with the tenets of qualitative research and this blog was intended to demonstrate a) that such different tenets exist and b) that we, as qualitative researchers, need to be better and making them explicit.
Greg,
I re-read your reply incl the SI and I’m still struggling to see understand your point of view regarding time scales.
In the SI for example you quote Jarraud as saying
“more temperature records were broken than in any other decade”
with the emphasis (italics) on the word “any”. Doesn’t that point to these temperature record being presented in the context of the longer-term trend (“than in *any* other decade”)? To me the answer to that would be a clear yes, but apparently that’s not the same to you. Could you clarify your position in that respect?
It may very well be that speakers mentioned (spatially and temporally) local events as *examples* of what climate change might mean to a person’s live, so yes, to make it more societally meaningful. I’m not challenging that. What I’m challenging is your premise that in doing so the IPCC speakers provided an incoherent picture of timescales, on the one hand presenting a decade’s worth of data as scientifically meaningful and on the other hand as not meaningful. The former, the warmest decade, was consistently put in the context of the long term trend, even in the quote that you mentioned yourself. So there was no such incoherence.
Hi Bart,
Thanks for the reply. Just as quick point of clarity – the emphasis on the word ‘any’ was Jarraud’s and was included simply because this is the transcription procedure that I use, not because we attach particular importance to that emphasis here. I don’t think this is essential to the point you’re making but it just seems important to note for the sake of clarity.
This issue in question is elaborated on in the original paper. I think it is important to note that the passage under discussion here was put into context in that original and there was never any intended suggestion that a discussion of trends was absent. In Supplementary Information B, for example, we say:
“…both Michel Jarraud and Rajendra Pachauri put particular emphasis upon the fact that, in Pachauri’s words, “the decade 2001 onwards having been the hottest, the warmest that we have seen” (L261-262). The claim that the decade 2001-2010 has been the warmest ‘we have seen’ was sometimes contextualised by noting a ‘trend’ for increasing temperatures within the last thirty years. Thomas Stocker, for example, stated that “the last three decades… has been successively warmer at the Earth’s surface” (L418-420), a claim also made, almost word for word, by Pacharui (L258-260).”
The short word limit of NCC obviously makes it difficult to present quotes of that length in the main document (this is another difficulty for the qualitative researcher!) but they were included in our submission and it seems important to note that they were.
Our key point in regard to this discussion comes several sentences later (again in Supp B) when we say:
“The focus, as Pachauri says, was upon ‘what we have seen’ rather than on invisible, abstracted, century long trends.”
And it is this seeing which is the important part of our argument. In the passage you mention (Michel Jarraud’s opening statement, L42-113 for the transcript) Jarraud talks about “raising sea levels, melting ice caps, glaciers… extreme events such as heat waves, droughts, floods” and states that “events that we are now witnessing on a regular basis. Many of the extremes of the last decade were unprecedented…” We entirely agree with you that context to these utterances is provided by noting the trend. What we suggest, however, is that the information contained here far exceeds that context; these observable events relate pretty obliquely to the thirty year trend; these are the *things* we have *seen* not the trends that we have subsequently become aware of. What is more, we argue that this excess plays a significant role in Jarraud’s call for us to act on climate change. As we say in our reply, Jarraud “leverages the witnessing of local meaningful events” to make his point. It is in the witnessing, the seeing, where we think the meaning is to be found. It is also this use of these witnessed, meaningful events which, we argue, leads to confusion when questions of uncertainty arise during the journalists’ discussion because, as Warren says above, they appear similar ‘prima facie’. As Warren says again:
“we are *not* saying that the two phenomena are the same, based on scientific knowledge. What we were saying is that beyond an expert climate science context, they appeared to be similar at first blush.”
This should also help re-confirm something we have said from the start; this paper does *not* question the scientific findings of the IPCC but is, rather, concerned with the successful communication of those findings.
As a further point of clarity, we do not say that Jarraud is wrong to evoke these images, we just state that he does and that it is a contributing cause to the confusion in the press conference. Neither do we suggest that trends are not likewise important, we just suggest that they are not the beginning and the end of the conversation.
I hope this helps in making clear what we were getting at,
Greg
Hi Greg,
Thanks, that clarifies your stance indeed.
Looks like we agree that speakers invoked observable events to make the abstract issue of climate change more meaningful. Neither you nor I have any beef with that. Actually, I think it makes sense to do so: if you want an absract message to stick you have to create some real-life references to make it stick. That’s communication 101.
We disagree that in doing so the IPCC became “incoherent as to ‘what counts’ as scientific evidence for AGW.” The observable events were used as an illustration, not as scientific evidence. In that sense you seem to have overinterpreted what these “observable events” were supposed to indicate.
Moreover, there was no incoherence w.r.t. timescales on the part of the IPCC, because the warmest decade was always put in context of the long-term trend. Since that’s the main message of your paper as I see it (based e.g. on the abstract), that’s where our disagreement comes from.
Thanks for engaging.
Regards,
Bart
Warren –
Thanks for the response.
==> “The data we refer to here is the transcript of the press conference.”
I was hoping you could elaborate – specifically as to describe the interesting patterns that you see in those data. Near as I can tell, your identification and quantification of those patterns is subjective, but your rhetoric seems to me to imply that there’s some objective identification of patterns.
==> “Of course, this is a reasonable point re Kahan etc. However, the point of the paper is *not* to demonstrate whether or not certain journalists demonstrated motivated reasoning.”
Of course. But my point is that you talk of interpreting the data to derive patterns when, I wonder, if the patterns that you’re talking about demonstrate something other than what you seem to assert. In other words, when you talk of “confusion,” how are you identifying and quantifying confusion? We can see a claim that the press conference was confusing or incoherent – but just because someone (i.e., David Rose) points to something and says it was confusing doesn’t make it so in some objective sense. Other possibilities are: (1) there’s something about that individual’s reasoning that is problematic…and that leads to their confusion whereas others were not likewise confused and, (2) that person claims “confusion” as a rhetorical device to advance their own agenda.
I don’t know if you’ve read the HotWhopper post on this topic. I would expect that there would be much there that you’d disagree with and I wouldn’t ask you to respond to the entire post…but I do wonder how you respond to the evidence she presents regarding whether or not reporters might have been “confused” by the press conference – evidence she interprets to indicate that there was no larger confusion.
==> “We do not wish to suggest that David Rose was particularly sympathetic to the IPCC prior to the press conference” and linking to two fairly polemical articles (neither of which I agree with btw).”
Well, there’s a pretty big gap between not being “particularly sympathetic to the IPCC” and pursuing an agenda to undermine the IPCC by claiming “confusion” as a rhetorical device to advance an anti-IPCC agenda. Yes, you don’t say that Rose was particularly sympathetic to the IPCC, but it seems to me to be problematic to generalize from his polemical articles that the IPCC was confusing or incoherent. If that’s not what you’re doing, then what other evidence or data are you using? I just don’t see how you can determine that the IPCC was confusing or incoherent without making it explicit what influences might affect or explain who is making such claims.
==> ” So a part of making climate change meaningful requires being attuned to these biases, as I would be very sceptical that one would be able to erase them.”
In that we are in agreement. But how could the IPCC maintain the integrity of their science and still make their communication meaning for David Rose, or other “skeptics” who may be committed to refuting any product of any “expertise” from anyone who lines up differently than them on the issue of risk posed by ACO2 emissions? That’s the nut to be cracked, no? What is the driving force behind what makes science about climate change meaningful for someone who may be heavily identified with confirming biases ala “cultural cognition?”
==> “Great, thanks! This is borne out by Jacobs et al’s Correspondence imho.”
Perhaps, but through discussion with one of the authors, it seems that his point of focus was not on the question of what comprises effective communication, but on whether or not, from a scientific perspective, what was stated at the IPCC presser was “incoherent.” Those are distinct issues.
==> “I’m sorry to read this, as this is definitely *not* my intention.”
I regret, then, my misunderstanding.
==> The literalism/lucidity conundrum doesn’t have any universally satisfactory answers see https://blogs.nottingham.ac.uk/makingsciencepublic/2012/11/14/short-circuiting-the-language-of-sandy-how-to-balance-literalism-and-lucidity/ .”
Thanks for the link. I don’t understand what you mean by the literalism/lucidity conundrum…perhaps after I look at your link I’ll have more to day.
==> “I *would* argue that attempts to depoliticise the climate policy debate (notably through the spurious fact/value distinction that the IPCC has been saddled with since birth) have laid the foundations for debates which have caused a lot of people to become stuck in intellectual mud over whether or not there has been a pause, what the exact value of climate sensitivity is, and quantifying the level of climate science consensus. The cultural and political questions about what we do with our climate knowledge dwarf these scientific curios imho.”
I agree. IMO, the arguments about those issues and myriad others (e.g., what comprises legitimate adjustments to temperature records, whether there’s such a thing as global average temperature, etc.) in effect, become a distraction from the more important issues: risk analysis and risk management. The difficulty in disentangling those two general themes is certainly understandable as they do seem, in some ways, to be inextricable. But a belief that we can smooth the road towards risk assessment and management by presenting scientific arguments about sensitivity, “consensus,” the “pause,” etc. seems to me to be contraindicated by what we know about the influences of motivated reasoning and cultural cognition.
We were struck that half of the questioners asked about ‘local’ phenomena; not just temporally local (the pause) but spatially local phenomena (regional climate impacts, local policy and local consumption). This was in contrast to the ‘global’ flavour of the press conference. We interpret this as the assembled journalists searching for greater meaning. In short, “What does climate change mean for my readers?” During the writing and review process, the paper became more narrowly focused on ‘the pause’ as that was the most prevalent local phenomena raised.
The notion of rhetoric is very interesting, which I hadn’t thought about before. Do you mean lines such as “Here we demonstrate that….” suggest an objective identification of patterns? If so, then I see what you mean. The NCC ‘Guide for Authors’ states that the introductory paragraph should include “the main results (introduced by “Here we show” or some equivalent phrase)”. In the future, we should give more thought as to whether this language is appropriate for interpretive, qualitative research which has very different traditions for what counts as rigorous research.
First, as far as I can see, the evidence presented is in terms of the press reports written afterwards. Our paper was very clear that our analysis was confined to the proceedings of the press conference *only*, which was streamed live on TV, not what happened afterwards. We *did* mention Rose’s article in our paper, but only because it made such specific reference to the press conference. Of course, it may well be that Rose would have found something negative to say anyway, but that’s not an excuse for giving him a ‘free hit’ as we argue occurred in the press conference. Press conferences are criminally under-researched; a paper which explains the apparent disjoint between the content of the press conference and the content of the media reports following would be very valuable. If the press conference did not inform the media articles, then why hold a press conference at all?
Second, it’s worth pointing out the actual meaning of the words we use (something we had to do in the SI of our Reply re ‘ignore’ and ‘repeatedly’). Confusion has a couple of meanings listed in the OED (free version at http://www.oxforddictionaries.com/definition/english/confusion): “uncertainty about what is happening” and “the state of being…unclear in one’s mind about something”. The fact that repeated questions were asked about ‘the pause’ suggests to me very strongly that there was uncertainty and a lack of clarity regarding its status and importance. Indeed, Alex Morales says himself he is “still not clear” why the IPCC talked about ‘the pause’ at all. Journalists (mainly) only got one question, why would they waste it on something that had already been asked satisfactorily?
As noted above, Rose was not the only one who asked these questions. The most prominent example is Alex Morales, who asked the following questions: “…you acknowledged that a fifteen year period is less relevant from looking at a climate point of view and thirty years is what you would normally look at. If that’s the case why did you even mention a fifteen year period in the summary for policymakers?” “And the first question was why include a fifteen year 1000 period in the summary for policymakers at all?” “Sorry could I just. I’m still not clear why you included it though. Is it because lots of, you were requested to include details about this and you felt it was something you needed to give a best explanation of?”
This is the $64m question which is the subject of our next paper. What kind of communication should the IPCC do, and for whom? This has been identified by the new Chair as a key topic. At one end of the spectrum, one could imagine a very dry, emotionless recital of the key points of the report. At the other end, one could imagine the Chair delivering a Jason Box-style exclamation http://www.salon.com/2014/08/06/climate_scientist_drops_the_f_bomb_after_startling_arctic_discovery/. What do we expect scientists to do? Should they rise above the fray, deliver the data and leave it up to the rest of society to sort out? Or should they get fully involved as activists for decisive action (whatever that might be). The answer is likely somewhere inbetween, but this unresolved tension for the IPCC seemed to be what the press conference speakers were wrestling with. Of course, Stephen Schneider identified this tension many years ago with his much (mis)quoted comment about the ethical bind of climate scientists in communication http://blogs.plos.org/retort/2013/08/13/a-correction-on-lomborg-and-schneiders-quotation/ (This is essentially what the literalism/lucidity post is about).
In 1999 Paul Edwards, who went on to write a fantastic book called “A Vast Machine” finished an article on uncertainty in climate science rather nicely. He says: “all parties engaged in climate science and politics should take careful note of how translations are made between scientific and political arenas. In politics, scientific uncertainty becomes a rhetorical resource which can and always will be employed by different interests in different ways. For opponents of immediate precautionary action, uncertainties… provide a time-worn rationale for shunting funds and attention back to basic research, or for denying any validity to climate change projections.” http://www.climateknowledge.org/figures/Rood_Climate_Change_AOSS480_Documents/Model_validation/Edwards_Climate_Data_Laden_Politics_SciCulture_1999.pdf
This is a key point, we think. It stresses that 1) uncertainties will always present and 2) *certain agents will always use those uncertainties for various purposes*.
Given that uncertainties are present in every science we think an acknowledgement of them needn’t undermine the case for action on climate change and a greater degree of honesty over this point would have helped substantially during the press conference. This is why we cite Chasing Ice in our Reply (and in comment above) which provides a very different, and in our view more meaningful, view of climate science. The NYT piece this week also provides a great, comprehensive view of the climate science enterprise: http://www.nytimes.com/interactive/2015/10/27/world/greenland-is-melting-away.html?_r=0
One of the claims in the original Hollin & Pearce article was that the IPCC presented an incoherent message, and yet both Greg and Warren’s responses to questions here seem to lack coherence (ironic?). The original claim seemed to be that the IPCC had fallen into a “certainty trap” by using one decadal indicator to stress their certainty about AGW, while dismissing another. One main argument in our response was to point out that these were not both decadal indicator, and – hence – there was no “certainty trap” or incoherent message. Both Greg and Warren’s responses here now seem to be suggesting that the issue was a communication failure, in that some people were confused. That may be true, but a communication issue is not the same as a certainty trap (or incoherent message). So, which one is it? Did the IPCC fall into a certainty trap, or did they simply present a message that some found confusing (or, possibly, neither).
Hi ATTP,
Your post here makes the same mis-reading as the original Correspondence. You state above:
This is absolutely *not* the claim made in the original paper nor at any time since. It is a fundamental mis-reading and something which we have noted on numerous occasions. In our reply, for instance, we state:
“[we] do not argue in our Letter that short-term events such as ‘the pause’ undermine any well-established certainty. Rather, we examine, first, the attempts of press conference speakers to make well-established certainty meaningful and, second, the resulting confusion among journalists…”
So we absolutely do *not* – *not* – say that the speakers used short term indicators to “stress their certainty” about AGW. That certainty is already incredibly well established; something which, again, we note *repeatedly*. The claim in the Letter is completely different: we say that speakers used short-term events to give AGW *meaning* – something which I think is clear from the above quotation.
I went back to Letter to make sure that this position was clearly laid out from the start and I firmly believe that it was. In the very first paragraph of the original paper, for instance, we say:
“Speakers at the press conference… attempted to make the documented level of certainty of AGW more meaningful to the public. Speakers attempted to communicate this through reference to short-term temperature increase.”
And:
“In this instance, the certainty trap was the result of the speakers’ failure to acknowledge tensions between scientific and public meanings.”
So the trap is clearly about attempts to increase meaning in public settings. Later in the paper we say the certainty trap stems from:
“The simultaneous reliance on some temporally local events to increase public meaning, and the dismissal of other similar events because they are uncertain.”
Again, this makes clear that this is about the utilization of short term events to *increase meaning*. So the trap is *absolutely 100% a communication issue*. This is why we’re studying a press conference. A press conference doesn’t seem to me a very sensible place to go examining scientific certainty but it seems like a pretty sensible place to examine attempts to give scientific findings public meaning. Again, a look at the method section makes this motivation clear:
“Publication in journals cannot be relied on as a means of communicating research outputs beyond the scientific community… Press conferences, therefore, are a means for scientists to reach non-specialist audiences and provide an important location for the study of science communication.”
So this is *explicitly* about science communication. Maybe it would’ve been better to call the paper “the IPCC’s meaning trap” (the title was a source of some debate with the journal) but from the position we’re in now I am genuinely unsure as to how we could make our position any clearer.
Briefly, addressing your second point:
I agree that this was your argument and we address precisely this question in the reply to Bart, above.
The abstract of your original paper said
Can you please explain to me how this is consistent with your claim now that you “absolutely do *not* – *not* – say that the speakers used short term indicators to “stress their certainty” about AGW.” The first sentence of your abstract has the word “certainty” in it, the second has “short-term” in it, and the the third has “similarly short” in it. I really do not see how you can claim you did not do this, when your abstract appears to claim exactly that.
To go through this sentence by sentence:
First sentence you’ve bolded here “…attempted to make the documented level of certainty of AGW more meaningful to the public.” “Documented certainty” clearly means that certainty is already established (i.e. in the report, not in the press conference). What was “attempted” in the press conference was to make “AGW more meaningful to the public”. So, what was at *stake* in the press conference was making an already certain AGW meaningful.
Second sentence you’ve bolded says: “communicate this through reference to short-term temperature increases.” So this sentence claims that the speakers attempt to communicate what was at stake (i.e. *the meaningfulness of AGW*) through reference to short term temperature increase. Making AGW certain is not what is at stake here, making it meaningful is.
Third sentence: “similarly short ‘pause’ in global temperature increase.” The claim is that *journalists* were confused that short-term events were being drawn upon by speakers (in order to make AGW meaningful) at one time and then other events, which seemed similar prima facie, were dismissed at another. This is the trap. Everything we have stated thus far is entirely consistent with this.
Except they did not do this (make it meaningful or certain) with reference to short-term temperature increases. If you’d said that it appeared as though they did this through reference to short term temperature increases, that might be true (since it would depend on how it was perceived). However, they did not actually do this, which is what you appear to be claiming. The decadal trends are NOT short term.
Except you’re now somewhat changing what your abstract actually says. You’re suggesting here that it appeared to the journalists that they had used some short-term events to make AGW meaningful, while dismissing other short-term events. Your abstract, however, claims that they actually did this, not that they appeared to do this.
This is the crux of the point I was actually getting at in my earlier comment today. There is a vast difference between claiming that the IPCC had actually done this, and suggesting that it appeared that they had done this. Your abstract seems to suggest the former, rather than the latter.
Hi Ken,
In the spirit of searching for consensus wherever possible, I’ll just comment on this claim:
Despite some differences between us apparently remaining, I note that in the comments above, your co-author Bart does agree with us on this point:
https://blogs.nottingham.ac.uk/makingsciencepublic/2015/10/23/methodological-clarity-required-when-publishing-social-science-in-natural-science-journals/#comment-1156282
Thanks, Warren
Hmmm, no I think he’s politely agreeing with something different. Bart should comment, but I don’t think he agrees that the IPCC used short-term temperature increases to communicate the meaningfulness of AGW.
I think Bart was agreeing that they did referr to observable events. My own take, though, is that even here they put those into the context of longer time intervals. For example (L82-83)
It’s hard to interpreted unprecedented as anything other than these events being something for which their no previous record.
interpreted -> interpret
for which their -> for which there is
I must admit that I’m now a bit confused (you might think that that is no great change from normal). In a recent comment, Greg stated that HP15 did *not* claim that the speakers used short-term indicators to stress their certainty about AGW. Warren now appears to be suggesting that HP15 did claim this and that Bart agrees. Have I misunderstood this?
Hi ATTP,
As I see it both Bart and I wrote about attempts to increase meaning rather than certainty (i.e., and as Bart states, “
”
Greg
Okay, then change it from “certainty” to “meaning”. The point, though, is that they did not focus on short-term temperature increases to do this. Even the discussion about “events” doesn’t really do this, given that “unprecedented” implies that it hasn’t been observed in the records, which aren’t short-term.
This, again, was addressed in the reply to Bart:
Except the journalists weren’t confused, as both their questions and their subsequent articles showed. And no event was dismissed. This is clear from the transcript. And, as ATTP pointed out, the claim in HP15 (echoing David Rose’s claim, echoing Richard Lindzen’s claim) was that the IPCC speakers were “incoherent” – but they weren’t. That claim of “incoherence” was baseless journalistic waffle on the part of David Rose.
This sidestepping and evasion of the flaws in HP15 is one of the more *incoherent* discussions I’ve read this week. Neither meaningful or certain. It’s as if there’s some other paper being discussed, not HP15. Maybe something that the authors wished, in retrospect, that they’d written instead of what they did write.
Hello Sou. As noted above, by ‘confusion’ we merely meant a ‘lack of understanding, uncertainty, the state of being…unclear’ (OED definition). The number of questions that were asked makes me sceptical that the message from the platform was as clear as you claim. https://blogs.nottingham.ac.uk/makingsciencepublic/2015/10/23/methodological-clarity-required-when-publishing-social-science-in-natural-science-journals/#comment-1158172 confusion
Regarding the subsequent press reports, the paper was clearly based within the parameters of the press conference *only*. We referred to the MoS article only as an example of an impact from the meaning/certainty relationship we identify. We did not intend the MoS article (or Rose) to be the focal point of the paper. It seems some have interpreted it this way.
I’m sorry you think we are sidestepping. You won’t be surprised to hear that I strongly disagree. I also disagree that we are talking as if it is a different paper. As we argued in our Reply, Jacobs et al’s Correspondence, in fact, largely missed the point of our original Letter. As we prepare our next paper on meaning and certainty in climate change, we will redouble our efforts to ensure that we communicate our ideas as clearly as possible.
This would seem rather ironic. Jacobs et al. was written by 6 people, 2 of whom have experience writing about climate science, 2 of whom are academics in the social sciences who focus on climate science, and 2 of whom are natural scientists, 1 of whom specialises in climate science. How did such a group manage to miss the point?
Since you’re suggesting that we were confused, maybe you can explain how
doesn’t mean “attempted to communicate this through reference to short-term temperature increases.
Warren –
Thanks for taking the time to write that response. It does help to clarify some of my misunderstandings.
However, I will say that there is an interesting irony playing out here. It is my impression, as Anders indicates is his impression also, that you have been saying that the scientists in the presser were confusing to some by virtue of being inconsistent: On the one hand they suggested that we can generalize from a short-term time frame to understand the long-term implications of continued aCO2 emissions, but on the other hand said that it is scientifically invalid to generalize from a short-term time frame to understand the long-term implications of continued ACO2 emissions.
So let’s say that in some objective sense I’m wrong, and that isn’t what you were saying. Even in that case, it might be more effective if we didn’t get caught in the weeds, trying to prove in some objective manner what the connotations of what you wrote were. After all, determining the connotations of what someone writes almost always involves some level of subjectivity. In the end, at least in a Western paradigm, communication acts are writer/speaker responsible. It is the speaker/writer’s responsibility to clarify what they intend to communicate. So maybe in this case, perhaps you could acknowledge that there is *uncertainty* associated with what you wrote, and what would be better is for you to revisit what you’ve written to clarify, clearly, that you agree that the IPCC *did not reference a decadal scale to convey meaning about the certainty of climate change at one point while rejecting the scientific meaningfulness of a decadal scale at another point.”
In essence, I think that you have done what I just suggested in the exchanges with me. I’m less interested in haggling about the precise interpretations of what you wrote and more interested in mutually beneficial good faith exchange.
That doesn’t mean, however, that I think that discussing in a more technical sense whether or not it’s accurate to say that the IPCC was “incoherent,” is useless. It *would* be useless if everyone just walks away from such a discussion simply further entrenched in the *positions* they held when the discussion started. However, in this case from reading the discussions I now have a better sense of why saying that the last decade is warmer than previous decades over an extended period of time is not the same thing, from a scientific perspective, as using a decadal scale to draw conclusions about the validity of AGW theory by reverse engineering from the “pause.” Perhaps you do as well?
In the end, I agree with what you wrote about the most useful way to integrate uncertainty into the discussion of climate change. That given, and given what I wrote above about “certainty” in communicative acts, I think you might revisit the following comment you made:
==> “…a greater degree of honesty over this point would have helped substantially during the press conference. ”
It seems to me that your message there is incoherent with what you said earlier: That you weren’t asserting that scientists were using scare-mongering. Perhaps saying something more along the lines of “a more proactive approach towards uncertainty would have helped” instead of “a greater degree of honesty..would have helped” would help reduce misunderstanding about connotations? And I would have hoped that you might have anticipated the potential disagreement about the connotations of “a greater degree of honesty” (assuming that you weren’t intentionally communicating a connotation that the scientists were lying for the purpose of scare-mongering).
Re ‘honesty’ yes that was a very poor choice of words dashed off in haste, as it implies greater agency on the part of the speakers. As we said in opening paragraph of the original paper, there was no sense *whatsoever* that the speakers were acting dishonestly. That is why we described a ‘trap’, as in something that they didn’t see coming. On your other point, of course there is uncertainty surrounding pretty much most statements one could make, including qualitative ones. Interpretive research always invites alternative interpretations of course, that is a good thing (although they don’t disprove the other one de facto). In our next (longer) paper we will further develop a theory on the meaning/certainty relationship, based on some positive and negative examples from climate change communications.
Just to add, in this thread, there are now new claims that the IPCC didn’t explain the different levels of certainty with the various findings. This, too, is contradicted by the transcript – as well as by the articles that journalists wrote immediately after the press conf.
Then there is the contradiction that scientists “dismissed” or “ignored” things, while at the same time pointing to questions on those same things that got a lengthy answer, and weren’t “dismissed” or “ignored”.
One more thing. The questions on regional projections were because it’s the first time that regional projections were included in the way they were. Seems odd to think that there’s something unusual in questions about regional climate. Scientists know that that’s the info that everyone wants – for practical reasons.
As noted above, the articles written afterwards were not part of our study. We looked only at the press conference itself, as a globally televised event in its own right. In the SI of our Reply, we note the dictionary meaning of ‘ignore’ and explain how this applies to the press conference questions. And, of course, questions about regional projections are not ‘odd’, we have never said anything like this anywhere. it is exactly what one would expect when people are searching for meaning within a highly globalised climate science.