This point is critical. A study can have a good methodology. But if nobody tries to replicate it, you have no chance of finding out that you are missing something critical. And with something as complex as people, "missing something critical" is something that we need to figure out.

And this isn't exactly a new problem for psychology. Feynman somewhat famously questioned a lot of classic research with http://neurotheory.columbia.edu/~ken/cargo_cult.html. I say somewhat famously because people in areas like physics have all read that essay. But in my experience psychologists are a little less inclined to read the criticism.

Even at the best of times, replication of results in a complex system is hard. See, for instance, http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj... for evidence that most published medical research is wrong.

But with psychology it is even worse. We lump people together by symptom, not by root cause. It is like lumping together people with migraines, head injuries, and caffeine headaches together. Then you find out that coffee beats a placebo for helping headaches, and give them all coffee!

You think I'm exaggerating? Read http://www.nimh.nih.gov/about/director/2013/transforming-dia... for verification that the NIH considers it a high priority research item to come up with classifications based on root causes, so that we have a chance of even being able to do useful research!

Yes, psychology has a lot of well-meaning people, and they stumble across a lot of interesting stuff. But it should be read with a critical eye, because the field as a whole does not perform reliably enough to compel our trust.