McIntyre: The deleted data from the “Hide the Decline” trick | Hacker News Reader

McIntyre: The deleted data from the “Hide the Decline” trick

wattsupwiththat.com

36 pointscwan38 comments

I'll probably never know what the data supports (although I'm admittedly skeptical), but I do know that discussing deleting emails and data to avoid responding to an FOI request is probably illegal, and definitely casts a cloud on everything else the participants have touched.

+1. I have lots of sympathy for emails being taken out of context, internal jargon looking weird when shown in public. But asking someone to delete email/data is just wrong and can't be explained away. I find it sad that there hasn't been an official response to that specific email thread. I also wonder whether it might be borderline illegal.

I didn't read the "ask somebody to delete data" part in that email? Also I think there have been official statements (I think the other article that was posted, about the "climate researcher who engages skeptics", mentioned the response).

The CRU have publicly asserted that they have not deleted emails or data.

"No record has been deleted, altered, or otherwise dealt with in any fashion with the intent of preventing the disclosure of all, or any part, of the requested information."

http://www.uea.ac.uk/mac/comm/media/press/2009/nov/homepagen...

Phil Jones, Wed Dec 3 13:57:09 2008: About 2 months ago I deleted loads of emails, so have very little - if anything at all.

http://www.eastangliaemails.com/emails.php?eid=940&filen...

Hat tip: Steve McIntyre.

http://camirror.wordpress.com/2009/11/24/denying-email-delet...

Deleting emails is not the same as deleting emails to prevent disclosure. I delete emails all the time; I presume everyone does. The email being quoted here doesn't say anything about why Jones "deleted loads of emails". You are inferring more than is warranted.

>"Mike,

Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment – minor family crisis.

Can you also email Gene and get him to do the same? I don’t have his new email address.

We will be getting Caspar to do likewise.

…

Cheers Phil"

http://camirror.wordpress.com/2009/11/25/willis-eschenbachs-...

Just a side question: I hardly ever delete emails (except for automatic reminders etc), and I always assumed that's the standard thing (I'm using gmail). But ionfish deletes emails, so I'm questioning my assumption.

Do you delete emails?

I never delete any emails. I find with several years of emails and a good search engine, it's like having a super memory.

Agreed, I've had email since I can remember and I'm only 21. Aside from them getting lost from formats, I have never deleted a genuine email.

Even with a piss poor search engine, it's better than my computer-dependent brain's attempt at memory. I've flat-out given up on remembering many things, I either have a note pad and pen or they end up on google Docs so I can retrieve them anywhere. (Edit: All I need is a good smartphone plan and Google Docs and Gmail can flat-out replace my need to remember anything)

I have in the past (when disk was a lot more expensive than it is now) deleted email that I was sure I'd never care about -- archives from projects that had concluded, for instance, and mostly consisted of "hey, can you come into my office for a minute" type stuff.

But I've never deleted sent messages. I pretty much have every message I've ever sent, minus a few here and there that perished due to hardware failure. This is mostly for CYA reasons but also just because it's a useful personal historical record.

I would be pretty suspicious if someone told me to go through my workstation and delete messages pertaining to a certain subject.

Although there are companies which, due to the legal climate they operate in, have information-destruction policies (basically keeping stuff for the amount of time legally required and NOT A DAY MORE), having a policy that's in effect all the time is a lot less suspect than requesting specific emails be destroyed.

I don't delete sent message either, but I do go through my inbox from time to time and clear out things I don't think I'll care about in the future. That may very well bite me in the ass later on, but that's why god invented backups.

I've never deleted a genuine email in my entire life. I've deleted thousands of emails, sometimes in one go, but I've never mentioned 'deleting loads of emails' it's "I deleted a shit load of spam today, it was a fucking boat load, you could feed a third world country!"

Perhaps yes he deleted a lot of spam, but no one on the planet considers them emails. Why would he be any different?

I try to go through and regularly delete emails I don't feel to be important. I archive the ones that I do want to keep, but the rest get deleted. I simply don't need to hold on to things like old conversations with my advisors about registration, emails from Facebook, Twitter, etc. containing updates, random emails amongst friends, and many more. I don't consider those to be spam, and some of them were definitely genuine email for the time, regardless of how you define something ambiguous like "genuine email", but I just don't feel the need to be a digital pack rat. There's a decent chance that I'm not the only one that feels this way.

a lot of places that are subject to FOI requests have policies in place to delete emails more than X months old (where X is given as part of the policy). Keeps costs down more than anything I think - FOI can be very expensive and time consuming (although it is a good thing).

From the email:

"The data are attached to this e-mail. They go from 1402 to 1995, although we usually stop the series in 1960 because of the recent non-temperature signal that is superimposed on the tree-ring data that we use. "

So who of the discussing people is well versed enough in tree-ring data to have an informed opinion on this? It sounds as if in recent times tree-ring data has to be read differently (which seems possible, other factors could affect growth, don't know).

Donald Keiller a plant physiologist at the University of Anglia wrote Briffa this October:

"1) The appropriateness of the statistical analyses employed 2) The reliance on the same small datasets in these multiple studies 3) The concept of "teleconnection" by which certain trees respond to the "Global Temperature Field", rather than local climate 4) The assumption that tree ring width and density are related to temperature in a linear manner.

Whilst I would not describe myself as an expert statistician, I do use inferential statistics routinely for both research and teaching and find difficulty in understanding the statistical rationale in these papers. As a plant physiologist I can say without hesitation that points 3 and 4 do not agree with the accepted science."

See link for full email thread: http://www.eastangliaemails.com/emails.php?eid=1064&file...

This is interesting too-

"I really wish I could be more positive about the Kyrgyzstan material, but I swear I pulled every trick out of my sleeve trying to milk something out of that. It was pretty funny though - I told Malcolm what you said about my possibly being too Graybill-like in evaluating the response functions - he laughed and said that's what he thought at first also. The data's tempting but there's too much variation even within stands. I don't think it'd be productive to try and juggle the chronology statistics any more than I already have - they just are what they are (that does sound Graybillian). I think I'll have to look for an option where I can let this little story go as it is.

Not having seen the sites I can only speculate, but I'd be optimistic if someone could get back there and spend more time collecting samples, particularly at the upper elevations.

Yeah, I doubt I'll be over your way anytime soon. Too bad, I'd like to get together with you and Ed for a beer or two. Probably someday though.

Cheers, Gary Gary Funkhouser Lab. of Tree-Ring Research The University of Arizona"

http://www.eastangliaemails.com/emails.php?eid=12&filena...

There really does need to be a full and detailed explanation of all this, immediately.

Doesn't that message support the reading "drat, I can't filter out enough noise to make it statistically significant, so I don't have anything to publish"? (Genuine question.)

It's in response to the climatologist trying to get him to publish anyway. He was being pressured to make his data fit.

Publish or perish. This all sounds very normal to me. I don't see references to making data fit, rather it sounds as if he is looking for a ways to extract some useful information from it.

Anyway, you know what: people believe what they want to believe. This discussion is completely pointless.

I am not well-versed in dendroclimatology, but I believe this issue is described here and is uncontroversial:

Briffa, K.R., Schweingruber, F.H., Jones, P.D., Osborn, T.J., Shiyatov, S.G. and Vaganov, E.A., 1998 "Reduced sensitivity of recent tree growth to temperature at high northern latitudes." Nature 391, 678-682 (R)

Basically, since tree-ring density is influenced by temperature, you can use it to reconstruct temperatures. I believe the data set in question uses maximum latewood density (MXD), which accorded well with instrumental temperature readings until 1960, at which point they begin to show a decline in temperature while thermometers show an increase. The proposed solution is simply: Don't use the reconstructed temperatures after 1960, since they're wrong.

We're being asked to believe that starting around 1960 tree rings "suddenly" no longer reflected local temperatures, so we shouldn't rely on them after 1960.

Doesn't this also imply that looking backward tree rings might be untrustworthy any earlier than some starting date? That in fact tree rings might work for a while, then vary for a while, then work again? How can anybody honestly claim that SOME tree ring evidence is usable and SOME is not -- and they know exactly which is which?

My bet would be the thermometers are wrong and the tree rings right. See, e.g.,

http://gallery.surfacestations.org/main.php?g2_itemId=831

The only way such an arbitrary cut-off date could be justified would be by explaining what caused the sudden divergence.

I would assume that if thermometer data is available, it is being used.

As for trusting tree ring data, this is an entirely different issue from this "Climate Gate" affair. That was, btw., an internal email, so nobody was supposed to trust anything. It was scientists exchanging some data. Surely it is allowed to LOOK at tree ring data.

I agree. We need an explanation _why_ tree ring data suddenly becomes unreliable in 1960, exactly when tree rings stopped showing global warming.

These scientists are asking us to trust their thermometer readings alone, starting at exactly the date when tree rings began disagree with their hypothesis. We need a very very good explanation for this incredible coincidence.

Here's your explanation: because tree rings represent an indirect measurement requiring a set of assumptions, while temperature measurements are direct measurements and require none. In the event of a conflict, trust the direct measurement over the indirect one.

There, that wasn't so hard, was it?

Here's my question, to anyone that knows: is the data set in question the only data set that the (alleged, I suppose) warming trend is observable in?

Or is this merely a nitpick at one particular set of data, while other independent ones exist that were generated in different ways?

I ask because in fight the over evolution, a lot of fuss has been made over a few very minor results that are seen as vulnerable in order to try to get people to ignore the big picture, which is clear as day (and doesn't depend at all on the individual results being picked at). While I realize not all AGW skeptics are anti-evolution, there's a big enough overlap to wonder if the same tactics or delusions might translate over...perhaps that's a cynical thought, but conservatives have earned no benefit of the doubt as regards scientific matters, so please forgive my prior bias in favor of the mainstream scientists - in every scientific field that I have personal experience in, the mainstream scientists these days are generally solid and honest, and certainly never fraudulent.

If this is really the only data set that shows a warming trend, then that's a lot more worrisome.

This is not the only data set that the warming trend is observable in.

[ In fact, some AGW skeptics don't deny a warming trend, but instead argue against human impact causing this warming trend (the A). ]

Nevertheless, here is what Real Climate says regarding the "hide the decline" bit (in more detail than their original post):

>>Declines” in the MXD record. This decline was hidden written up in Nature in 1998 (http://www.nature.com/nature/journal/v391/n6668/abs/391678a0...) where the authors suggested not using the post 1960 data. Their actual programs (in IDL script), unsurprisingly warn against using post 1960 data. Added: Note that the ‘hide the decline’ comment was made in 1999 – 10 years ago, and has no connection whatsoever to more recent instrumental records.

So, every climate record produced in recent years has no connection to this data. And there's lots of these.

Your evolution analogy is apt. Many (very smart) people would like to pretend this simply isn't happening or isn't possible.

the real nightmare here is that this is going to cast a huge shadow on legitimate climate research -- and science in general -- for many years to come. these douchebags need to rot in jail for a long time.

It's a bit extreme to put them in jail. I too think their actions are deplorable, but jail? We didn't even put the criminals behind the financial collapse in jail, why put morally reprehensible scientists in jail? I'd say the financial criminals caused more problems than the climate team.

Correct me if I'm wrong, and I'm not a British citizen so my understanding of the nature of publicly funded work may be a bit different, but if they are colluding to change the results of a study paid for with public monies and destroy evidence requested by an official inquiry, isn't that sufficient cause for criminal charges? The coordinated aspect of their fraud -- exemplified in the leaked emails -- adds a damning, worrisome aspect to the whole affair.

On a sociological level, this is a very dangerous situation. The impact of human endeavor and pollution on global climate is almost impossible to judge on a local personal level, so the public must have faith in the scientists we elect to study this problem.

Criminal charges would send a strong, immediate signal to scientists in all areas of research: check your personal opinions at the door when processing data. We are all human but if we allow this kind of thing to creep into all areas of research we risk undermining the very scientific underpinnings of our modern secular approach to public policy and, ultimately, the governance of our natural resources.

Scientific research must remain balanced.

As an aside: the people behind the economic crisis are not in jail, but they should be. Just because they have so far escaped the reach of justice doesn't mean we should lower our standards for other blatant crimes just because they are of the white collar variety.

> if they are colluding to change the results of a study paid for with public monies

What study? Are you referring to the fact that at some point they drew a graph that updated data extrapolated from tree ring data with actual data from thermometers, and that that graph may or may not have actually been published? How long a jail term does that merit?

> destroy evidence requested by an official inquiry

What official inquiry? They talked about deleting emails they considered private. How long in jail for that?

> Criminal charges would send a strong, immediate signal to scientists in all areas of research

On our backward little island we don't yet throw people in jail on the basis of their academic research. I'm sure we'll get there soon.

As someone who has done some experimental work, I can appreciate how emails like this sound bad but may actually not be.

With every set of data, there is a lot of additional information necessary for interpreting it. I'm sure that if someone is just given my raw data, for example, it is open to interpretation -- however, ALSO given the "metadata" (which is often in my lengthy, informal lab journal), the rationale behind my interpretation is more clear ...

If I had to explain every little nuance to amateurs and explain why their interpretations are incorrect, I would certainly not have much time left in my day!

And therefore you would start deleting emails, ask others to delete emails, claim that original data has been lost when it has not in fact been lost, etc?

What would you do if you have a hypothesis and you see that from a particular date onwards the data doesn't support your hypothesis? I think you would try to find out why that is.

And if your work directly influences policy and taxes and living standards you might have to provide an explanation that "amateurs" can understand because "we the amateurs" are the same as "we the people" mentioned there:

http://topics.law.cornell.edu/constitution/preamble

I could see myself starting emails if somebody hacked my accounted and started publishing my private correspondence.

Obviously it's not desirable to have email accounts hacked.

But what has emerged so far is work related correspondence of public employees working on publicly funded projects. And the content of those emails is about trying to hide data that is supposed to be subject to public scrutiny and peer review.

Depending on what kind of organisation you work for, deleting that kind of "private correspondence" may constitute a crime.

It has been suggested that some of the data these people were trying to hide was subject to a FOI request (http://en.wikipedia.org/wiki/Freedom_of_Information_Act_2000). I don't know if that's actually the case, but if it is we may see prosecutions in the future.

I didn't follow this in detail, so far I have mostly seen some words taken out of context that can mean one thing or another. Personally I prefer not to judge words that are taken out of context.

In that sense, I suppose an actual prosecution would be a good thing, to end all the senseless speculation.

Reply on news.ycombinator.com