Hacker News Reader

Comment by huitzitziltzin | Hacker News Reader

On the contrary, failure to replicate should definitely reduce your posterior probability over the original findings being either true OR useful. True findings should replicate. There are complexities - maybe you can't perfectly recreate the original conditions, especially in non-experimental settings - but broadly speaking replication is the right standard to use.

View on HN

The presumption here is that it was a true replication, and that's pretty damn hard to accomplish in the soft sciences. It matters, just not nearly as much or as conclusively as in something like physics.

If a result is incredibly hard to replicate, that's a good indicator that the effect is either so niche, or so small, that it's not actually telling you any details of the world worth talking about.

If 'priming' only works every wet Tuesday in February, after someone has had a cheeseburger with no sauce and took their Grandma to the bingo, perhaps there's some _actual_ interesting artefact explaining it...or it could just be random noise.

A lot of the studies involved do replicate, just with reduced effects.

But the presumption here is that the same experiment is even being run a second time, and my point is that many studies who have the stated goal of replication do so in ways that I find hard to justify.

So even when a study claims to "fail to replicate", I don't think we should just take that on face value.

Reply on news.ycombinator.com