CRU Admits to Dumping Raw Climate Data
timesonline.co.uk
timesonline.co.uk
If you adjust data to normalise it due to the differing global climatic effects of El Nino or La Nina then that's perfectly reasonable, if someone can corroborate what you did and if you did it right. However with no original data set, and apparently no mention of how or why the data was modified the way it was leaves it completely unverifiable and might as well be entirely fabricated.
The best explanation, until proven otherwise, is that somebody goofed.
Professor Jones: Just sent loads of station data to Scott. Make sure he documents everything better this time ! And don’t leave stuff lying around on ftp sites - you never know who is trawling them. The two MMs [Patrick notes: he is referring to McIntyre and McKitrick] have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone.
The rest of that email reads:
Does your similar act in the US force you to respond to enquiries within 20 days? - our does ! The UK works on precedents, so the first request will test it. We also have a data protection act, which I will hide behind.
Why then, would they bother to mail it around, as they did in the quoted email?
I suspect there are some other reasons.
This makes me glad again that there is some kind of legal system in most civilized countries. So many people seem to be ready to form a lynch mob at short notice.
I don't know any of the people involved, but I really despise lynch mobs. Still, I know I shouldn't bother with this.
Of course, if it turns out that it was illegal for them to accept the data under that license in the first place, then it might be more understandable.
Not wanting to deal with the MMs does not imply guilty, just as not wanting to discuss moot points with creationists does not imply evolution theory is wrong. At some point maybe you decide your energy is better spent elsewhere.
Just playing devils advocate... I still feel there is too much context missing.
...and if the FOI won you wouldn't be able to get data anymore.
I don't understand what you're getting at with your second paragraph.
Under that license, right.
I don't understand what you're getting at with your second paragraph.
What I mean is that if government documents are subject to FOI requests, then it seems possible that there's a law about accepting data under a license which would conflict with that. I don't know that there is, but it seems possible. That would be an understandable reason having no bearing on the climate debate to delete rather than provide the data.
With technology as it is and science as it is and has been for hundreds of years, this seems like an extraordinary goof. I could see a goof being losing a block of data, but all of it? Why was destruction of this data even an option or consideration? How expensive and difficult is it to keep it? I suspect there are numerous other organizations that would store it if they were given copies of it.
"Purposely" might be a bit extreme but it was more than a simple goof.
No, I believe malice is really the only explanation. Raw data allows reproducibility and intelligent debate.
Not that I know that the same applies to the data in question here, just saying. I still wait for the full story.
Your example does not demonstrate this point. By taking the temperature readings in degrees C, you, in fact, are taking the "base data". Of course, you also should record the equipment that you are using to take that data. The height of mercury in your thermometer can be determined by the the brand of thermometer (including the Manufacturer and Model #), and the temperature reading given. Most scientific peer-reviewed research articles are very specific about the equipment being used.
I don't see how? It is a matter of how you define "the base of the data". My point is that there is a calculation on the measurement, and only the result of the calculation is recorded. Hence the base data is discarded.
We don't know what calculations were done on the climate date in question here. But it seems at least possible that the calculation is also trivial (for climate researchers anyway, could be some standard procedure), just as to us converting heights to degrees is trivial.
Recording the type of thermometer is another matter - likewise the climate researchers will probably record where their measurements were taken, which implies the equipment used.
Edit: of course if the calculations are reversible is another issue. I see now that is the point you made with the thermometer (calculate height from make and degrees). Well, it was just an example, and we don't even know yet if the calculations in question here are not reversible. I suppose if they calculated an average they are not. But I also think there are probably many cases where a measurement is just taking an average (probably even the height of quicksilver is an average in a way...).
??? If the calculations are reversible, there would be no argument. Everyone would know the base data.
I think you are kind of pushing it with your quicksilver example; I don't see how taking a single measurement is "averaging". If you mean to say that there is some built-in variance - well, that's true. But the fact is the variance itself can be established by other scientists if they know the model of thermometer that is being used. This is the essence of repeatability!
Now, imagine that a scientist had many temperature readings, and then they did some kind of complex calculation which involved regression analysis, statistical modeling, etc. These are the types of calculations that could easily have errors. Without the initial recordings, the results are unverifiable. That doesn't make them necessarily useless, but I wouldn't lean on those kinds of results.
So basically there is no evidence the earth is warming? Because evidence based on unreleased data is worthless.
If I can't reproduce your work, you are a crackpot (isn't that the definition of junk science?) I can't reproduce their work, so they are crackpots. (Or deliberate frauds.)
Did they even record their methodology on how they adjusted the numbers? Maybe we can undo that methodology?
If they didn't record even that, then they are incompetent. And I see no reason to pay attention to incompetents.
What if someone made a simple arithmetic error? Without the raw data, and the methodology of adjustment how could we ever know?
Their conclusions are worthless, and unless there are others (are there?), then there is no evidence the earth is warming.
Unfortunately this is true. As someone who has never doubted that global warming was man made (yes, those last leaked emails were private conversations and held no smoking gun whatsoever. People were reading into one term used ten years ago to sex up a graph), this is embarrassing. This, more than any other thing has made me reconsider my views.
The problem is that glaciers and polar ice are melting at unprecedented rates, this is unarguable. The earth is getting warming in those areas and it is effecting the sea levels and biodiversity of the planet. How do we tell if it's man made? What can we do to stop it? Remember this doesn't prove that global warming isn't real.
What happened here? They threw away their copy of the raw data. That data still exists (they gathered it from public repositories), and the methods of the paper describe what they did. You could reconstruct the results.
Essentially, the "skeptics" are getting worked up over what amounts to throwing away the lab notebooks for a paper published two decades ago. Even if you assume malfeasance on the part of these authors (an assumption that does not appear to be justified), you'd have to disregard the results of thousands of other papers to leap to the (wild) conclusion that "there is no evidence the earth is warming."
If what you say is true, that's really good. It's probably not true, judging from the comments in the source code, where the guy laments on his inability to verify the very existence of some of these stations. That's not to say the stations he wanted, didn't, at some point, probably exist, it's just that time passes and the world moves on, people retire, and filing cabinets go to the salvation army.
Anyone who has spent any time around scientists knows them to be a conservative and naturally skeptical group of people. The "scientific establishment" -- if there is such a thing -- is powerfully opposed to new ideas. The climate change debate was a controversial new idea thirty years ago, but after three decades of research and debate, it's been as close to universally accepted as anything ever is in the world of science. The people who would have you believe that climate change is a conspiracy amongst rogue scientists either do not understand how science works, or are purposefully dissembling to advance their cause.
fallacious "skeptical" arguments
The climate change debate was a controversial new idea thirty years ago
Just call it global warming and admit that 30 years ago, climatologists were wrong. Don't try to obfuscate their lack of perfection (nobody expects perfection anyway) with this "climate change" nonsense of "we knew it all along, see, we predicted climate change 30 years ago!"
No, not really. There's a difference between being skeptical and using fallacies and rhetoric to cause confusion. I disagree with people who do the latter in the guise of "skepticism".
"See, this revisionist stuff drives me crazy. 30 years ago, it was called "global cooling". Now, sophisticated punters call it "climate change", instead of the less sophisticated and more embarrassing flip-flop of 'global warming'."
What part of "the climate change debate was a controversial new idea" don't you understand? When presented the theory, scientists didn't believe it. They debated many different theories -- global cooling, global warming, temperature cycles, etc. -- and eventually, the data led them to a theory. It's not "revisionist" when new data leads you to change your hypothesis. It's science.
They debated many different theories --
global cooling, global warming, temperature cycles, etc.
So you have to go back and recollect the info from all these sources (not an insignificant undertaking, though mostly do-able), actually give a reason for the "corrections", and then create models that fit that data, and then issue new predictions.
Without starting over from raw data and following all the way to the conclusions reached about man made global warming there is no scientific proof that man causes global warming. None of it is valid until you can show how that "correction" was made. And the loss of this data means they will never 100% be able to show that.
no evidence the earth is warming.
The earth is definitely warming. We are exiting an ice-age. There has been a conscious effort, I believe, to equate global warming with man-made global warming. They are not the same thing, even if used interchangeably by the media.
The "hockey stick" graph appears to shows that the earth is warming faster now than it was earlier. It's this "steeper slope" that we're debating, and that may have been manipulated. The general public, however, has had the debate obfuscated for them into "is it warming, or isn't it?"
If Darwin's original notebooks were lost, there would not cease to be evidence for evolution. There are many thousands of papers on climate change. A loss of the raw data sets for one paper does not change the conclusions drawn by the others.
No, it's not. First, no reputable journal allows re-publication of the same data set -- after thousands of publications, it's just silly to suggest that it's all based on the same data. Second, even if you don't believe that other scientists would prevent the re-publication of the same data, one only has to peruse the IPCC report to see that there's a lot more than just the "hockey stick" graph and computer models.
Which is a strange read. "Looks like the National Review was way out front on this one," is not something I expected to say to myself today.
I'm not trying to allege bias, just making the observation that stories that don't fit into pre-existing media narratives take a lot of push to get out there. Usually it's the fringe media that keeps kicking and kicking until somebody finally picks it up. If I remember correctly, the National Enquirer was running stories of John Edwards' love child for months before anybody else would pick it up. (And love or hate Edwards, it was a real story that deserved national attention)
I wonder if blogging has changed any of this? Sure would be great to read a study on how this type of anti-narrative story breaks today versus 20 years ago.
1) The emails were acquired through an illegal hack, which makes them a highly tainted source for journalists who aren't pushing an agenda. Naturally journalists who are pushing the opposite agenda will ignore it, as those who who are skeptics will readily jump on them.
2) This leak was initially trumpeted as a the end of AGW theory, and has since turned out to be nothing more than scientists being douchey and unprofessional wrt their critics. Smoke, but no fire. There's an important lesson in marketing here: Make sure you set appropriate consumer expectations.
3) (US only) It's technically foreign news, and the American media generally doesn't do foreign news unless it pertains to our various wars or some 3rd world dictator says something bad about America.
Long story short, this would be covered in the MSM if: 1) The emails were acquired through a whistleblower with legal protection 2) it wasn't over-sold in the blogosphere the moment it broke 3) US Media actually gave a crap about what happens outside of America's borders that doesn't involve a military escapade.
Which means it probably is not hacker news.
(Also, consider whether we want to see both sides of this issue posting their stories - if the anti-warming folks post a lot of these, the other guys are sure to want to see their side all over the site as well. I think it belongs elsewhere)
The discussion between "the anti-warming folks" and "the other guys" has been ongoing for some time on this site: http://searchyc.com/submissions/global+warming
Have you always taken the position that global warming is not hacker news? Or has something changed so that is was hacker news before and isn't any longer?
I have a few bête noir topics I try to moderate away, too, but I tend to draw the line at explicit endorsements of one side over another in a tribal conflict. I can see how it would make sense to just excise tribal issues and have done, though.
It would be nice if there was a flag-like mechanism for the community to turn off comments on articles like this one. It seems that if the karma incentive was reduced on political articles, people would have less incentive to post the thoughtless ones.
This is backwards. Usenet had political flamewars despite having no karma.
People post to political threads because it makes them feel good to stick up for their particular causes. Emotions that reward us for signaling our allegiances and general loyalty are one of the cognitive adaptations that allow our species to solve the Prisoner's Dilemma to benefit from cooperation, specialization, and trade.
Start reading Robin Hanson: http://www.overcomingbias.com
or Comeuppance: http://www.amazon.com/Comeuppance-Altruistic-Punishment-Biol...
Flamewars on Usenet were hard to ignore, and easy to propagate. How would people start flamewars on HN if they can't comment?
Yes, a crime was committed, but you can't put the genie back in the bottle. There's precedent, too. From Wikipedia: The Pentagon Papers... were a top-secret United States Department of Defense history of the United States' involvement in Vietnam... [They] first surfaced on the front page of the New York Times in 1971.
...
This leak was initially trumpeted as the end of AGW theory, and has since turned out to be scientists being douchey and unprofessional wrt their critics...
But the facts remain: The data are gone. The wingnuts knew it. The media didn't report on it, in large part because the only people talking about it were wingnuts. Doesn't that make you feel a little uneasy?
That being said, I think there's a questionable assumption being made that these in fact are hacked illegally obtained emails. From CBS: http://www.cbsnews.com/blogs/2009/11/24/taking_liberties/ent... - "It's not clear how the files were leaked. One theory says that a malicious hacker slipped into East Anglia's network and snatched thousands of documents. Another says that the files had already been assembled in response to a Freedom of Information request and, immediately after it was denied, a whistleblower decided to disclose them. (Lending credence to that theory is the fact that no personal e-mail messages unrelated to climate change appear to have been leaked.)"
Two take home messages. One: we never lost raw data. Two: we never lost the scripts for applying corrections and were always in the position of being able to replicate our work. Shortage of disk space meant that we sometimes had to do this, so it had to work.
I suspect that journalists are slow to realise that deleting the raw data is a very odd thing to do. You delete your data when you have no students, have lost faith in your research and are committed to giving up and leaving the field. Do you delete your raw data if you are battling to save the planet? No.
I'll allege bias for you, then. There is no conceivable way that you could produce documentary evidence of a Republican saying, in as many words, "Destroy this so that it can't get used for a Freedom of Information (Act) request" and not have that be the front page of the NYT the next day. I don't care if he was the Deputy Undersecretary of the Department of Who Gives A Care, that sort of thing is normally blood in the water. It attracts front page attention, massive follow-up coverage, and Pulitzer prizes. When it targets the right people.
Anyway, your comparison is apples and oranges. Political scandals will always carry more weight in the public eye than science ones, if for no other reason than the fact that their hands are directly on the levers of power, whereas scientists just use a (relatively) small amount of taxpayer money compared to what is doled out in the halls of political power. And their impact on our daily lives is significantly less direct than even a mid-level staffer.
Personally, I suspect that the AGW thesis probably stands given that it's more than this handful of apparently corrupt scientists. That being said, where does the science stand if we take out these suspect models and the ensuing research that was based on them? Given these scandals, why is there not more of a movement to make these climate models and data in effect open source?
The level of secrecy, defensiveness and arrogance as shown by example in the Scientific American is especially troubling when you realize that practically all this research was publicly funded. Why are there non disclosure agreements on the data to begin with? (A real question - does anyone know?)
I think there is a risk there. Much of the work in the field is reworking and resplicing old results -- for a graphic illustration of this, see McIntyre's graph of which tree-ring series are used in which papers.
http://noconsensus.files.wordpress.com/2009/03/slide18.jpg?w...
Primary data are hard and expensive to collect. Computer munging on existing data is, relatively, cheap to produce -- you only need grad students, pfft, they work for ramen. So the papers tend to gradually build up more series, more stats, more regressions on the same existing data sources. Then you get into computer modeling, which takes the assumptions gleaned from the 43rd reworking of the source data and combines it with the 27th reworking of other source data to predict that in scenario N where X assumption holds...
What if we're building a skyscraper on a foundation of rotten timber?
P.S. That being said, where does the science stand if we take out these suspect models and the ensuing research that was based on them?
I think the scientists are very, very reluctant to do that. Look at the amount of remixing that has to be done for these papers. If you were to draw a graph of all the results in this field, connect all papers via citations (or a less promiscuous method of your choice, such as "Did I use their numbers for a calculation of importance?"), color one early result in the field red, and then floodfill -- I think it is quite likely that the whole discipline goes red. This is presumably why, when McIntyre et al have demonstrated problems in earlier research, they always get fobbed off with "Pfft, yeah, maybe, but we had 200 data sources and that was only two or three of them."
P.P.S. There is no incentive for a scientist to go back and check their own papers for errors introduced by a paper or data source they relied on suddenly going bad. Heads they spent a lot of work and their paper survives scrutiny: good news, but defending results from 1993 didn't get them a grant now did it. Tails, they find an error, can't correct it, and admit so. BAM, loss of professional reputation, funding, etc. And then they get to the really sticky business: calling up all their friends who cited them for the "Bob, got a bit of bad news for you..." conversation.
What perhaps frightens/annoys me the most is thinking that perhaps politically, some of these people are so invested in AGW that any new data is irrelevant so we will end up spending hundreds of billions if not trillions trying to deal with a problem that may end up not even to exist. Personally I think a lot of the changes in technology and energy usage favor reductions in greenhouse emissions and this should happen regardless, but some of these other massive costs...
I guess we'll see in coming days which narrative wins out - that nothing is done and is as irrelevant as the media thinks or the mainstream media begins covering the story grudgingly only because competing forces in governments (ie congressional investigations / investigations) make it impossible not to.
The media was highly critical of GWB during his whole presidency. He was portrayed as a stuttering moron.
This is in complete contrast to the current love-fest with Obama.
(I am not American btw, just making an observation).
From a tree-shaded plateau facing Mt. Kenya, the worshipers gaze anxiously at its melting ice cap and wonder: Is God dead?...The scientific community is divided over the causes of melting ice caps in Africa. But many experts believe the retreating snow on Mt. Kenya is one of the continent's clearest examples of climate change and global warming.
There are two layers of bias here, one obvious and one subtle. Obviously, "The scientific community is divided over the causes of melting ice caps in Africa" means that some experts think the melting is due to global warming, and some experts think it's not. Long-time observers have come to expect the Times (and the MSM generally) to choose the pro-global warming interpretation every time.
The subtle bias is that the Times wrote this article at all.
None of this means that human-caused GW isn't a fact -- what it means is that proving it and quantifying it are turning out to be very difficult. What the email scam shows is that people on the 'right' side of the debate (ie proponents of human-caused change) at some point went over a bit to the dark side, and decided that "failure is not an option" -- ends justify means, etc.
This saddens me because I don't want 'my' side (liberal, rational, forward-thinking) to be guilty of the same anti-science bullshit that GWB and crew were. If science becomes completely politicized, it will cease to be science. Where will be then?
humm..... citeseer with evidence / result tracking....
Boo-Hoo!
See http://www.realclimate.org/index.php/archives/2009/11/the-cr...