Most Published Research Findings Are False–But Little Replication Goes Long Way

journals.plos.org

44 points·gwern··3 comments

I would like to see replication handled with the introduction of a new type of journal: one that only publishes studies that have been previously published but are now backed up by additional, confirming studies. Maybe two replications would be enough.

It would be even better when paired with a journal of studies failing to reject the null hypothesis. The first journal could then base the publishing threshold on statistical power of the group, rather than a simple replication count.

It's a bit trickier than the title says. By a "false" finding, they mean "a finding fails to yield the same result in at least one other study." But there's nothing wrong with failing a replication. See cancer risk factors research papers: they regularly yield results with opposite signs.

Rather, we should talk about "more research is needed to get a representative sample of studies of a phenomena."

> See cancer risk factors research papers: they regularly yield results with opposite signs.

You'll have to expand on that - when I think of 'cancer risk factors', I think of data-dredging epidemiological analyses which are predictably wrong by the logic of hypothesis-testing, or I think of biochemical studies which blatantly overfit by failing to cross-validate or keep holdout samples, and lack of replication seems like it has the usual meaning.

Reply on news.ycombinator.com