First results from psychology’s largest reproducibility test
nature.com
nature.com
Edit: Just to expand a bit --
Suppose I flip a fair coin 100 times, and publish each result as its own god-given Truth. Now somebody comes along and questions my 100 Truths, so they flip a coin 100 times; The expectation value is that 50 of my Truths reproduce. Naturally there is a sigma (of about 10 I think), so the reproducibility study shows 50 ± 10.
The linked reproducibility study is right around expection - sigma. I realize that's numerology, but kinda funny. Nevertheless, my original point: The reproducibility study should be run several times to understand the random nature of the results. (Do the same studies reproduce their results? Or is it a different 50-ish results?)
The Reproducibility Project calculated the sample sizes necessary in advance, so if the effects are the size the original researchers claim, they'd have good power to detect them.
Their power could be worse than they expect, though; pioneering studies tend to overestimate effect sizes, because their sample sizes are too small and they filter for statistical significance. I call the problem truth inflation: http://www.statisticsdonewrong.com/regression.html#truth-inf...
Anyway: understanding the random nature of results is exactly the job of statistics, and the Reproducibility Project researchers are being very careful with their statistics.
Hal Pashler, “A lot of working scientists assume that if it’s published, it’s right,” he says. “This makes it hard to dismiss that there are still a lot of false positives in the literature.”
> · Recent hand-wringing over failed replications in social psychology is largely pointless, because unsuccessful experiments have no meaningful scientific value.
> · Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way. Unless direct replications are conducted by flawless experimenters, nothing interesting can be learned from them.
{edit: read the link. It's scary to read this stuff coming from a science PhD.}
Other psychologists have realised that there's a "crisis of replication" in psychology. But it's not just psychology, it affects other sciences (albeit to varying degrees) too.
http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
When something goes wrong in either of those fields, they always have an explanation - they're never wrong. Economy didn't lift off in 2015 Q1? It was the weather. But we know the weather affects all Q1, can't they factor that into the prediction for Q1 numbers? No, because they want to be able to use the weather excuse.
Follow what they say and you will start to see the pattern of excuses. Every time they are sure something will work; then it doesn't work; then they claim they knew all along it wouldn't work, but this new solution will definitely work. Rinse, repeat.
1. The results, when filtered through the replicability guidelines rendered a clear verdict - 61 to 39 against.
2. The question of "how closely did the findings resemble the original study?" flips the findings. Moderately similar findings are the majority - 58 to 42 in the other direction.
How can you have a study that has "virtually identical" findings that doesn't replicate the original?
(I am not a frequentist, so my bias may be showing through)
* "Of the 61 non-replicated studies, scientists classed 24 as producing findings at least “moderately similar” to those of the original experiments, even though they did not meet pre-established criteria"
* "Daniele Fanelli, who studies bias and scientific misconduct at Stanford University in California, says ... psychology does not necessarily lag behind ... other sciences. ... earlier studies have suggested that reproducibility rates in cancer biology and drug discovery could be even lower"