Wait -- so apparently it's acceptable in economics to publish big important papers with broad-reaching public policy implications without releasing the data on which the paper was based? That would never fly in the sciences.
Or would it?
http://climateaudit.files.wordpress.com/2009/12/mcintyre-grl...
Regardless of what side you come down on w.r.t. this particular issue (and I realize it's a sensitive one), the fact that global policy was influenced by a paper that used secret, buggy Fortran code to manipulate a data set is extremely concerning. This is not good science.
Publishing data and source code should be a requirement these days.
1) big problems in the climate science community with regard to data and replicability (see climateaudit for more than you could ever want)
2) massive statistical problems with reconstructions. See Mcshane and Wyner 2011.
"The test in science is whether findings can be replicated using different data and methods. More than two dozen reconstructions, using various statistical methods and combinations of proxy records, have supported the broad consensus shown in the original 1998 hockey-stick graph"
Meanwhile, everyone proceeded as if it were fact.
We need to raise the bar of skepticism and with it the burden of proof. Show me your raw data. Show me your code for manipulating that data into the final result. I'm honestly surprised that publishing data + paper + a source code repository isn't required in order to be taken seriously these days.
"7) Basically then the MM05 criticism is simply about whether selected N. American tree rings should have been included, not that there was a mathematical flaw?
Yes."
"8) So does this all matter?
No."
You should see how bad it is in my field of study - biomechanics. It's a pure black box. We have total cronyism, where one reviewer has disproportional power since he was the first to publish much of the modern stuff. Therefore, people will do anything to have him review their papers, and his papers are pushed through despite what many feel are seriously glaring errors in methodology and collection.
This is the walled garden crap that turns me off to "peer-reviewed research."
For a much fuzzier field like economics, where the line between knowledge and opinion is extremely blurry, this "bending over backwards" to show how you might be wrong should be applied 100x .
This is in part why I am a programmer now and why my Poli Sci degree is sitting in a drawer.
More traditional poli-sci is scientific in the sense of the social sciences, which has a fairly long history of epistemological debate I'm only vaguely familiar with. I think I would probably call it scientific in a certain sense, but maybe a different word is needed. I'd group it vaguely with disciplines like anthropology, archaeology, and linguistics as areas with quite a bit of methodological diversity, but still a more empirical orientation than you find in the humanities. Part of the issue is that there is data, but how to interpret the data is complex ("there is no such thing as raw data"). Though for mostly institutional reasons there are some people who are more philosophers or historians who also happen to be in poli-sci departments.
The American politics professor had an academic pedigree and insisted that Political Science could and should be a hard science backed up by facts and numbers. In his papers you need statistics, studies, and graphs. He focused very much on the methodology of your research and how well you could back up your claims with solid evidence.
The theoretical professor appreciated hard facts but it was much more important to him that you had a well-reasoned argument, and as _delirium said, would much rather read a 10-page paper full of epistemological debate than one twice that filled with detailed Bayesian analysis.
The international politics professor didn't care because he had tenure.
It's pretty common in medical research.
If a conclusion is only provable with one specific data set, then it can hardly be considered a universal objective truth of nature.
By analogy: if I drop a hammer and tell you that gravitational acceleration is 9.8 meters per second squared, you don't need to come over to my house and borrow my hammer to test it.
Sharing data and code is helpful for error checking, as it was in this case.
I don't know about the particular stat the article is talking about, but they give plenty of data here.