Comments in ClimateGate source code: unambiguous smoking gun
wattsupwiththat.com
wattsupwiththat.com
The most damning excepts from the leaked email are the following:
http://www.powerlineblog.com/archives/2009/11/024996.php In this one, they delete emails rather than respond to a FOIA request
http://www.powerlineblog.com/archives/2009/11/024995.php In which they lie to the NYTimes in response to a question of data cherry picking. They tell the NYTimes that they looked into the criticisms and that the criticisms were completely unfounded. But behind the scenes they were emailing each other that they could not explain the results, and in fact, other scientists had found the criticisms to be quite valid.
It's another piece of circumstantial evidence. By itself, it wouldn't mean too much. Considering everything else, it doesn't reflect well upon them.
You need to be very, very careful when normalizing or adjusting a complex system, and you need to be extremely clear that you're adjusting and how you're doing it.
A lot of normalizing/adjusting based on a certain factor is to make data fit a predetermined conclusion. There's some that are generally accepted - like adjusting for inflation - but even that can be done many different ways with different goals to serve different agendas. Adjusting data without disclaiming how you did it is a pretty big no-no and casts a lot of doubt on someone's results.
I am a skeptic as well, but the way these emails and code comments are analysed is entirely meaningless.
However, it goes to show how incredibly stupid it is for scientists to not be as transparent as possible in the first place.
We should be able to conduct our scientific research without constant fear of an "audit" by Steven McIntyre; without having to weigh every word we write in every email we send to our scientific colleagues. ... I am unwilling to submit to this McCarthy-style investigation of my scientific research... I will continue to refuse such data requests in the future. Nor will I provide McIntyre with computer programs, email correspondence, etc. I feel very strongly about these issues. We should not be coerced by the scientific equivalent of a playground bully. I will be consulting LLNL's Legal Affairs Office in order to determine how the DOE and LLNL should respond to any FOI requests that we receive from McIntyre.
They shouldn't be surprised now to find that they're being tarred.
Of course, that doesn't help us extract the truth of the situation. But it's a pretty clear object lesson to other scientists.
The author of the post says something like, "see, these programmer people leave comments in code to remind themselves later of what they really mean the code does. Armed with this knowledge, we can read some comments and find out what's really going on!"
No, dude, I'm a programmer and you can't just take random chunks of comments from code and authoritatively state what you think they mean.
It's also hard to take seriously a blog plastered with ads for $49 "Make-your-own-free-energy"-kits. The comments are a freak show of conspiracy theorists.
This sounds backwards. We calibrate against the instrumental record up to 1960, we know that there is increasing divergence post 1960, thus should we presume we can have greater trust for the previous eras where we have no instrumental record?
Wouldn't the onus be on explaining the divergence? I haven't seen anyone else accepting a divergence while at the same time claiming that the previous record is accurate. It seems more standard to downplay the divergence rather than to accept it.
Could you reference some sources that discuss the divergence?
I must have misunderstood something there.
http://www.ldeo.columbia.edu/res/fac/trl/downloads/Publicati... (PDF) is a paper that specifically addresses a lot of the concerns about temperature prediction using tree rings. In particular, I'd like to quote the following section as it pertains directly to your question:
"A number of tree-ring series indicate a divergence between tree growth and temperature at some northern sites in recent decades (e.g. Briffa et al. 1995, Jacoby and D'Arrigo 1995, Briffa et al. 1998, Vaganov et al. 1999, Barber et al. 2000). Theories for the cause (s) of this observed divergence, which may vary from site to site, include decreased temperature sensitivity due to warmer temperatures, drought stress, increased winter snowmelt and ozone effects. This divergence needs to be considered to avoid bias in dendroclimatic reconstructions; however it is not present everywhere. For example, temperature-sensitive elevational treeline sites in Mongolia and the European Alps exhibit dramatic growth increases in recent decades (D'Arrigo et al. 2001, Buntgen et al. 2005)."
http://www.ldeo.columbia.edu/res/fac/trl/downloads/Publicati... (PDF) is an earlier paper referenced in the above that also discusses the problem.
http://icecap.us/images/uploads/Loehle_Divergence_CC.pdf (PDF) is a paper by someone that's more skeptical, it goes into a fair bit of detail about the possible causes, mostly to argue against them.
Of course, we're getting a bit away from the actual topic. We know about the post-1960ish divergence. We know that it's a good idea to exclude that data from any kind of analysis, which isn't a problem because we have accurate modern measurements from that period to use instead. There may be other periods of inaccuracy - but if there are, they haven't been discovered, and there is a substantial amount of overlap between these reconstructions and our actual instrumental record during the calibration period. In other words, it's the best reconstruction we have at the moment, and like any scientific theory, it's subject to change if better evidence were to come along. Unfortunately, the skeptics seem to be more interested in politicizing the debate rather than finding that better evidence themselves.
None of this really changes what the comment says, which was basically "Don't use this to plot past 1960, because we know that part needs corrections". I don't see an issue with that. The wording could be better, but it's a useful note to have, to make sure that the data isn't misused. I don't think it would be necessary to say "Oh yeah, the rest is just a theory, it could be wrong too" - everyone who was supposed to have access to the source knew that. It comes with the territory. The problem is when internal stuff is leaked and that territory changes - a random person reads that comment, completely misses the tentative scientific context and thinks that the programmer is asserting the absolute correctness of everything he doesn't single out as wrong.
Edited to add in a (PDF) tag that I forgot.
Factors like that are certainly going to have an impact on the recent tree record.
there is a substantial amount of overlap between these reconstructions and our actual instrumental record during the calibration period
From memory, I don't think this is quite the right conclusion. Rather, the instrumental record is used to calibrate the reconstruction, so the overlap approaches 100%. Worse, the accusations are that the correlation is even better because any trees not matching the temperature record have been eliminated as non-performing! I don't think either of these papers shows accuracy of reconstruction on an out-of-sample set for which there exists an outside measurement. I'll recheck, though.
None of this really changes what the comment says, which was basically "Don't use this to plot past 1960, because we know that part needs corrections".
I don't disagree here. In the absence of some overriding context, that seems like a valid reading of the comment.
As for the calibration period, you're right, there are definitely some issues there, and that's a problem with the entire temperature reconstruction approach. We only have reliable instrumental records back to the middle of the 19th century, so it's hard to test any kind of reconstruction on a large scale.
Personally, I don't put much faith in the reconstructions themselves - I just think that it's important for people to understand the context around these experiments, since there's so much blatantly false information going around. It's hard enough for those of us who aren't climatologists to keep up with the information without people screaming bloody murder over a few lines of comments that they don't even understand.
I mean have you seen some of the arguments against things like evolution by 'lay people?' The correct course of action here is to get them to produce an explanation, then evaluate whether it's reasonable or not.
Edit: Maybe I should mention that there are plenty of things a doctor could do wrong that anyone could be able to call him/her out on, but there are many things where the subtleties may confuse.
Obviously it's not as mature a field as, say, medicine, but refusing to believe that any of the results are at all valid is just being wilfully ignorant.
i.e. Get some climate scientists to weight in on this and evaluate what they have to say on it rather than just looking at some of the evidence on your own and drawing your own conclusions.
I'm not saying to have a climate scientist give you a 'yes/no' answer to "Is this a fruad?" and then blindly trust their answer. I'm saying that someone in the field of climatology should be able to explain some of this away in a reasoned and understandable manner with facts that can be verified if this isn't some sort of fraud.
Declaring something a fraud by looking at snippets of emails and comments from source files out of context is not the way of rational debate. I highly doubt that you or many of the people rushing to label this as "100% of everything said about Global Warming since the beginning of time is now false" have done more than read snippets from blog posts.
The physics behind global warming makes a lot of sense. The warming models aren't pure physics, however, and require feedback effects. Now, are the models right about 6 degrees of warming in the next century? I don't have a clue because the historical climate reconstructions are so shitily done. I have no idea if the temperature anomaly over the last 30 years exceeds historical bounds for standard deviation. I have no idea what the magnitude of the Medieval Warm Period was.
This is an important question. A trillion dollar question. We need to throw the hopped-up climate scientists out, fools who were lucky to be in the field when it became important, and get some real intelligence working on this problem. Cause this ain't it.
For my part, and I am educated in the sciences (though not environmental ones, to the extent they can be considered "science"), I have no freaking idea what the true state of affairs is, and I doubt that anyone has much valid knowledge in this area. I don't trust any of the data that I see because a) historical inference is really, really difficult, and I'm very unimpressed by how it's been done, and b) there's so much incentive on each side to mangle the data to fit the conclusion. I don't have any faith in the environmental researchers to fairly report what the data tells them, but I'm just as skeptical about the bias of people arguing against them, aligned as they tend to be with the idiots and organizations that fight so hard against evolution.
Just to be clear, I absolutely believe that there are valid theoretical reasons to consider the possibility of (human caused) global warming; the models are more than enough to show that it's a real possibility. It's only the claim that it's already been observed (and that it's significant, reversible, etc.) that I'm skeptical about.
However, I think that strategically speaking, the environmentalism movement's increased focus on global warming is a big mistake because it gives anti-environmentalists a fixed target to nibble away at, and they're doing a pretty good job of causing doubt. Wasn't it obvious enough that spewing pollutants into the air is bad? Why not focus on that, like everyone used to? Instead, the environmentalists have taken on the burden of proving that global warming is man-made, catastrophic, and reversible; until then, the burden was on the anti-environmentalists to prove that it was safe to pollute as much as they wanted, which was pretty much an impossible task.
I worry that far more important environmental concerns than carbon output are being ignored because of the excessive focus on global warming, and this shift in attention could be truly catastrophic.
It's true that we've become obsessively focused on global warming as of late, although I would place at least as much blame on the media for that. In the past couple years, though, some groups have been doing a better job of saying "BTW clean energy will also create jobs and improve our national security." The WE campaign is one such example.
When I ship code to a client you can bet the ambiguities have been removed. Thorny parts have defensible explanations.
I've been in that situation before, as has anyone who's had to work with real-world data.
This hack episode seems really scary to me. Getting angry at the computer, your suppliers, et cetra is all part of the development process --- as is writing angry comments and internal emails. It's a much less disruptive way to vent the frustration that comes with the job than punching your monitor.[1]
I've not seen anything anybody's posted that doesn't happen at every development organization in the world. I really feel for those guys and the firestorm they must be in now. And to be frank. I hope the criminals who broke into their computers are captured and prosecuted.
[1] The nicest thing about the old CRTs is that they could take a good smack or two.
~A Jedi hand wave~
"These are not the smoking guns your are looking for."
Honest question: What projects are you referring to?
It's my understanding ( and please correct me if I'm wrong ) that the dataset in question is one of the core underpinnings of the findings of the UN's IPCC. And I'm not up-to-date on the policy implications in your country, unless you live in Canada. But in chilly Canada, I see our local carbon-tax lobby use the global warming meme in their effort to impose heavy taxes on our industrial base. Hence, the "de-industrialization" angle.
I'm concerned that this might involve more power-grabbing than scientific concern for our future. Hence the interest in properly designed and documented datasets.