That said, finding this was a big deal: http://camirror.wordpress.com/2009/11/26/new-the-deleted-dat...
The linked article also claims that the NOAA "deleted" the post-1960 data from a data set (briffa2001jgr3.txt) put together for an article that doesn't use these data. In other words, what really happened, is that Briffa and his collaborators created a data set only containing the data they used, and this derived data set was archived by NOAA. That's hardly nefarious or even the least bit worrisome.
Now to the heart of the issue: Why was the post-1960 not used after 1998/2000? Why do all of the e-mails talk about "hiding the decline" and the code talk about stopping in 1960 to "avoid the decline"? The answer, it turns out, is right in the very Briffa 1998 paper your link mentions (but fails to cite). It's title is Reduced sensitivity of recent tree growth to temperature at high northern latitudes and it's about how temperatures reconstructed using Briffa's MXD (maximum latewood density) data diverge from actual temperature readings after 1960. The implication of the word "reconstructed" is very important to understand: Briffa's MXD data don't show an actual decline in temperature. They show a decline in a particular measurement of tree-ring density that had, until 1960, been a good indicator of actual temperature. All that's going on here is that data that is known to be inaccurate is not being used.
It's no big deal.
This was the first time that the data was available. As McIntyre points out in the top of his post, the decline was discussed in the 1998 and 2001 Briffa papers.
Of course, scientists could not actually evaluate this because the data was not released. But that's not the issue.
The issue is, as McIntyre also points out, that the IPCC 2001 graph (and later graphs) simply truncated the climate reconstruction data in 1960 and pasted the (more accurate) direct temperature station measurement data onto the graph after 1960.
While this was done it plain sight, it's still lousy scientific practice. Why? After all, shouldn't more accurate data be used? The problem is that comparison of temperature reconstructions with direct measurement tells you a lot about the accuracy of your reconstruction.
In this case it tells you a lot about the inaccuracy of the reconstruction and the sorts of signals that are missed.
I'm sorry, the paleoclimate reconstructions available in the literature are not convincing. They often use dodgy statistical techniques, mixing data gathering steps with data correlation steps to increase apparent agreement with the instrument record. There have been huge problems with releasing data and there are often bizarre choices made in proxy record selection.
Without good paleoclimate reconstructions, it's hard to know how far outside (or inside) the historical standard deviation for temperature we are. I happen to think it's an important question to have answered. A trillion dollar question.
Any data that you receive from another scientist with the intention of reproducing their experiment is fundamentally tainted and violates the scientific method.
Please don't argue against straw men, it's annoying.
Scientists are normally obligated to release all data. It's the standard in most other fields (e.g., biology, econometrics).
Running the same data analysis on existing data doesn't count as reproducing the experiment. Running a different analysis on the same data is, however, doing new science.
In contrast, failing to release the data does count as "violating the scientific method".
Something completely unknown is making tree rings too small after 1960. And if you correlate your tree ring data to post-1960 temperatures it makes historical temperature look way too high.
Bizarro-Briffa sent a similar letter to Bizarro-Nature warning of an unknown divergence problem and saying not to correlate tree rings to pre-1960 temperatures because it makes historical temperature look way too low.
Since we do not know why the temperature has not correctly correlated over the last 50s years, we can't really know when else that has been the case, can we?
So why do we rely on tree rings for any period?
They are proven to be unreliable over a signifcant period, and we don't know if or how often that's also true of the past.
Thus, either (a) it has already been done, or (b) it is more difficult than you would expect for non-obvious reasons. So what is your point? Have you even bothered to find out if anyone has done it or not?
The standard of evidence to show fault is yours. Assuming that papers published in freaking NATURE have sound statistics is a reasonable thing for a layman to do.
I thought that's what you were asserting. My assumption is that scientists have correlated tree ring data with other sources and found it to be unreliable. But of course, neither my nor your assumptions about what's possible and impossible in this area are worth a damn, because we __don't know what we're talking about__. That is my point.
Also, all this talk about fallacies, ad homonim, etc. etc., is very "internet". Especially since you only seem to have a vague grasp of what these words actually mean. Do you think we could maybe leave all that stuff out and have a serious discussion?
That or you go look for the papers or the reason the papers have not been written.
This game is old. The standard of proof to demonstrate the method is flawed based on poor correlations is on your side. The method to do the correlation is public knowledge. The data to correlate is public knowledge. R is open source. Go forth and correlate.
You're confused about something: the post you're replying to clearly wasn't accusing an entire scientific field of engaging in "completely terrible" statistics. The poster was just questioning whether the comparison you were suggesting was feasible given our current (rather imperfect) knowledge of temperature changes over the last few hundred years. It may not be a matter of just getting two sets of figures and comparing them.
This is a clear claim that climate science researchers have not done basic statistics when presenting new models. This claim has been substantiated by nothing except you requesting more and more detailed analysis from non-experts, which you discount every time we provide it because you assert that you would only listen to an expert.
Ironically, you are actually failing to respect the authority of experts, because the discussion is fundamentally an accusation that experts are failing to do basic statistics. You are accusing us of not being experts so you can't listen to us in the defense of the experts you're attacking by proxy.
I seriously fucking doubt you understood any of this. But it doesn't matter, I won't return. I have no tolerance for people who argue by asking questions and petulantly demanding unreasonable proof while providing absolutely nothing of their own.
Thanks for ruining hacker news.
My problem here is that I'm not sure which metaphorical goalposts you are referring to. By the way, constantly talking in terms of "fallacies" (especially these informal ones, as opposed to actual fallacies of propositional or predicate logic) just makes you sound like a petulant 15 year old. (Ad hominem -- I'll do that one for you.) Don't refer me to a Wikipedia page on the informal fallacy of the month; address what I'm saying instead. Otherwise, you may be guilty of the fallacy of Missing the Goddamn Point Because You're Looking For Fallacies Instead of Reading What the Other Guy has Written.
Getting back to the issue at hand, I can only suggest that you (a) reread the thread and (b) calm down. I am not accusing any scientists of failing to do basic statistics, and neither was anyone else. I am merely expressing my awesome ignorance of all things related to climate science, and suggesting that perhaps some of the other posters here are about as ignorant of these issues as I am, but lacking a suitable degree of caution in their statements.
Good luck with leaving HN. Sadly, I very much doubt that you will find any discussion sites worthy of your superior intellect.
Climate research is research. It is not some know-it alls claiming one thing or another. It is making up theories and testing and aligning them with available data. It is trying to get new sources for data. And so on. It is an ongoing process.
Science never KNOWS anything. It only assigns weights and likelihoods.
We don't know if all history of earth is just an illusion, because we could just be a computer simulation that started 10 seconds ago.
So we will never KNOW if tree ring data at all times before 1960 correlated with temperature. All we can do is to collect evidence, make up theories and compare them with reality.
Disclaimer: I am not a climate researcher myself.
Trees are complex organisms whose growth relies on many factors in a complicated relationship. Just as a simple example, if you have a long, hot drought, do you expect tree growth to be more affected by the temperature or the lack of moisture? The scientists must control for these kinds of factors to isolate the temperature signal. But when you're dealing with the interactions of complex systems, new factors can emerge and the methods need to change to keep up.