Comment by SeanLuke | Hacker News Reader

Comment by SeanLuke | Hacker News Reader

I think the fundamental problem is that for every claim yielding a positive result there are many more, perhaps infinitely more, related claims yielding negative results.

Positive result claim: the sun comes up in the morning.

Negative result claims: the sun moves sideways in the morning. The sun was always there. The sun peeks up in the morning and immediately goes back down. And so on.

Positive result claim: aspirin is an effective pain reliever.

Negative result claims: eating sawdust is an effective pain reliever. Snorting water is an effective pain reliever. Crystal Healing is an effective pain reliever. Etc.

Because there are so many negative results, it's trivial to construct an experiment which produces one. So why should that be published?

Negative results should be published when people in the community are asking that question, or have a wrong belief in the answer (hence the replication crisis). But if nobody cares about the question, it's hard to argue for why a given negative result would be preferred over any other negative result for purposes of publication.

None of these are negative results in the sense of being a 'null' hypothesis?

In the language of hypothesis testing you have your null and alternative hypotheses.

So for alternative hypothesis that the sun comes up in the morning, the null hypothesis would simply be that the sun does not come up in the morning.

Each of the negative results, reads to me like a separate 'alternative' hypothesis.

Sure they are.

So let's say I claim that the sun goes in a circle in the sky in the morning. The null hypothesis is that it doesn't do that. Perform experiment. Null hypothesis wins. Write up paper! This is a negative result.

The point is that for every result where the alternative hypothesis wins, there are a massive, if not infinite, number of results where the null hypothesis will win. Are these publishable?

The idea is that some null hypotheses being true is actually interesting because it challenges an assumed belief. From the first paragraph of the article, the immediate feedback from the postdoc's supervisor was 'you did it wrong [because everyone knows that fish do like warmer water]'.

> It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.

> The idea is that some null hypotheses being true is actually interesting because it challenges an assumed belief.

??? As I had said originally, that's one of the primary situations where a negative result should be published.

But the huge, huge, huge majority of negative results are trivial and uninteresting. Thus the fundamental issue with negative results is that you have to provide rather more compelling justification for why such results should be published.

Yeah, I agree with your first point, but maybe misunderstood your reply? If there's nothing "surprising" about the result, it's not interesting, so not publishable. The article's first example, however, did seem to be surprising to the researcher's community, so it should have been published.

Sure. What I said, or had meant to say, was in reply to people complaining that there was some kind of cartel against negative results. Rather, what we're seeing is just the natural, if unfortunate, response to the basic problem with negative results as a whole. You can't just treat them as the same as positive results: because of their numerosity they require unusual justification for their publication.

Reply on news.ycombinator.com