I disagree with Turing and Kahneman regarding the strength of statistical evidence

This was also at the start of the cold war, where the US suspected that the USSR was funding millions to ESP research.

One of the most publicized projects was the Stargate Project: Even though a statistically significant effect has been observed in the laboratory, it remains unclear whether the existence of a paranormal phenomenon, remote viewing, has been demonstrated.

I think Turing was closer to the machine learning camp than the statistics camp. On that note I'll quote a competitor in the MLSP 2014 Schizophrenia Detection Challenge:

To the people that are going to write papers for this one...

What really strikes me is the fact that stats fail really hard in this problem. I have a couple of 2-variable combinations that score around 0.87 on training set with logistic regression and a couple of 3-variable combinations with training AUC 0.9 (ish). All results were "statistically significant" at 0.001 (not even 0.01) . I have tried the same selections with SAS, SPSS, R and scikit (with regularization) . All results are consistent (and similar) with all packages, yet again they scored around 0.5 (random) in public and private leaderboard. This makes me think about all the PhDs' thesis and medical science papers I've seen being carried out on mickey mouse sets , claiming statistical significance gives credibility to their findings ... Is machine learning more reliable than stats? I say, if you can't predict it consistently on a hold out set, then you got nothing whatever the t,F,Chi-sq distributions say. KazAnova - https://www.kaggle.com/users/111640/kazanova

The chapter on priming in "Thinking Fast Thinking Slow" completely ruined the book for me. The very premise of the book is (paraphrased): "I will teach you to overcome bias and think rationally" and then here is an entire chapter that should make anyone with a critical eye strongly question the research. Even if one is ignorant of the issues these experiments have with reproducability the experiment design itself is terrible, and there seems to be no way to draw the wild conclusion that are drawn. Not to mention that if priming actually worked as claimed you would see it become a hotter topic among marketers than SEO. I fail to see how a world expert on cognitive bias can fail to question such an obvious fault in a topic that he covers in his own book.

[0] http://www.nature.com/news/nobel-laureate-challenges-psychol...

http://lesswrong.com/lw/1ib/parapsychology_the_control_group...

Anyways, this article singles our Turing and Kahneman but it seems this is a fairly common pitfall/mistake among, well, almost everyone who uses statistics.

The other thing researchers do is look for effect X, fail to find it, but do find effect Y. They don't publish about X because journals aren't interested in null results. They do publish about Y because, well, money.

BBC Radio 4 did a reasonable programme about this: http://www.bbc.co.uk/programmes/b04f9r4k

  Unfortunately the statistical evidence, at least for telepathy, is overwhelming.

  You have no choice but to accept that the major conclusions of these studies are true.

Statistics are great for providing evidence to counteract our intuition, and show empirical holes in logic. That said, they should (like most everything else) always be considered provisional truths as opposed to final truths. Experiments can be misdesigned. Stats can be misinterpreted. And (God forbid!) statistics can be misinterpreted.

In all seriousness though, having worked analyzing data in research, one overwhelming issue is that statisticians are often only involved in the analysis and not the experimental design resulting in studies that are plagued with noise and other confounding factors.

It is hard to know how this would effect a study of ESP etc since even a batch of non randomly selected people who could reliably do this would be significant but non the less replication has failed.

it's still up to our brains to try to make that last leap between data and prediction, which is something our brains are exceedingly good at, if necessarily flawed (in a schrödinger's cat kind of way).

gödel (among others) helped us find the limits of logical systems like this. i've mentioned it before (in https://news.ycombinator.com/item?id=8169025) but i'm really looking forward to the development of non-logic based math (quantum computing?) that's more akin to how our brains work and allows for multiple, conflicting "truths" to exist simultaneously. our consciousness lives in the rational world but the rest of our brains seems to be quite content with an ambiguous, entangled reality.

"Wagenmakers, E.-J., Wetzels, R., Borsboom, D., Kievit, R., & van der Maas, H. L. J. (in press). A skeptical eye on psi. In May, E., & Marwaha, S. (Eds.), Extrasensory Perception: Support, Skepticism, and Science. ABC-CLIO. NB. I was requested to remove this preprint temporarily, until the publisher has given explicit consent to publish it here. If you seek a preprint please send me an Email."

And I'll note for the record that many of the papers by Eric-Jan Wagenmakers and his co-authors continue to be available on his website,[1] whether published yet or not, but please be sure that he is allowed to carry out his agreements with publishers so that those papers stay available. The work of that group of authors is very important and will do much to improve the science of psychology.

Gelman also comments favorably in his blog post about the work of Uri Simonsohn and his colleagues, who have devoted much thought to the issue of "p-hacking." Simonsohn is a professor of psychology with a better than average understanding of statistics. He and his colleagues are concerned about making scientific papers more reliable. Many of the interesting issues brought up by the comments on the article kindly submitted here become much more clear after reading Simonsohn's various articles[2] about p values and what they mean, and other aspects of interpreting published scientific research.

Simonsohn provides an abstract (which links to a full, free download of a funny, thought-provoking paper)[3] with a "twenty-one word solution" to some of the practices most likely to make psychology research papers unreliable. He has a whole site devoted to avoiding "p-hacking,"[4] an all too common practice in science that can be detected by statistical tests. You can use the p-curve software on that site for your own investigations into p values found in published research.

He also has a paper on evaluating replication results[5] (an issue we discuss from time to time here on Hacker News) with more specific tips on that issue.

"Abstract: "When does a replication attempt fail? The most common standard is: when it obtains p>.05. I begin here by evaluating this standard in the context of three published replication attempts, involving investigations of the embodiment of morality, the endowment effect, and weather effects on life satisfaction, concluding the standard has unacceptable problems. I then describe similarly unacceptable problems associated with standards that rely on effect-size comparisons between original and replication results. Finally, I propose a new standard: Replication attempts fail when their results indicate that the effect, if it exists at all, is too small to have been detected by the original study. This new standard (1) circumvents the problems associated with existing standards, (2) arrives at intuitively compelling interpretations of existing replication results, and (3) suggests a simple sample size requirement for replication attempts: 2.5 times the original sample."

[1] http://ejwagenmakers.com/papers.html

[2] http://opim.wharton.upenn.edu/~uws/

[3] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2160588

[4] http://www.p-curve.com/

[5] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879