Hacker News Reader

Comment by scott_s | Hacker News Reader

The obvious fix is to (a) require raw data to be published; (b) require journals to accept papers before the experiment is performed, with the advance paper including a specification of what statistics were selected in advance to be run on the results; (c) raising the standard "significance" level to p<0.0001; and (d) junking all the damned overcomplicated status-seeking impressive nonsense of classical statistics and going to simple understandable Bayesian likelihoods.

Suggestion b is both radical and very thought provoking. Which, at this point, I expect from Yudkowsky.

View on HN

I think in some fields achieving (c) is almost impossible - the n number needed would be too expensive for most labs working with animal models, for instance. (b) would be interesting, especially if it led to a culture where "failed" results are still worth publishing because they disprove something. I think (a) is the one worth focusing on first, since that seems the most achievable and would let you back-apply some of the other improvements at a later date.

Interesting. The idea that null results are not worth publishing has since the very beginning struck me as one of the most fundamentally flawed ideas in science. Interestingly, it seems to be very domain-dependent. In my field (astrophysics), null results are fairly frequently published, but I've heard that in other fields it's totally impossible to do.

I've run across one or two compiler optimization papers where the conclusion given was that the proposed technique didn't work out so well, but on the whole, it seems it applies there as well. I agree that it's a problem -- if a null result is not published, other people will probably waste years making the same mistakes.

Yeah, the wasted repeated effort was what I initially thought about, too. The argument about how compilations of results will be systematically skewed by certain results not being published is perhaps even more persuasive, because it doesn't just lead to wasted effort but to incorrect results.

Reply on news.ycombinator.com