Most published research results are false

Math and CS are better off than experimental sciences. Theoretical paper often contain errors, but these don't usually invalidate the conclusions.

In experimental sciences, hard sciences like physics do have a lower error rate than sciences that rely heavily on statistics. But that's changing. See the second page of this article: http://www-stat.stanford.edu/~ckirby/brad/papers/2005NEWMode... The author argues that "Nature has gotten more tight-fisted with modern physicists" and that physics is now more dependent on statistics.

> Note well. This is for clinical research.

My own experience is that there is much constant questioning about published clinical research.

In addition, decision-making of clinicians based on clinical research is tempered with other considerations, sometimes quite complex. Stephen Paulker commented in response to the original paper:

"From the perspective of an epidemiologist or a statistician, the relevant question is whether the study's hypothesis is true - i.e., is the probability of H1 greater than 0.5. For clinicians and their patients, the relevant question is whether a particular strategy should be followed in an individual patient or a subset of similar patients. That decision (or recommendation to the patient) will depend on the pre-study likelihood of benefit in that patient and on the relative magnitude of benefits and risks of that strategy, if the diagnosis in that patient is uncertain. For many such decisions, the "more likely true than false" criterion may not be the best decision rule. For serious diseases and treatments of only modest risk, post-study probabilities of considerably less than 0.5 may be sufficient to justify treatment."

From what I've seen, I wouldn't say CS is immune to this at all. CS seems to only be accepting papers that illustrate some big win.

In my short PhD career, I've an article in the drawer where the result was inconclusive. This result, in and of itself, was fairly valuable. It spoke much about all the ways things went wrong. However, wrong was not acceptable for publication, it had to be right. Much hand-wringing was done to identify the bits that could be pulled out from it, polished, and then submitted (I was not especially pleased about this). The paper was, rightly, rejected, but all the valuable findings are still in my drawer. The pressure to produce Good findings rather than Bad findings does not meet my expectation of scientific pursuit.

I took great joy in seeing a history of science exhibition recently, and reading the notebooks of people like Newton and Darwin, who they themselves spend a lot of time writing out all the things that went wrong, all the concerns they have about their "results", and general humbleness. It speaks volumes that the greatest scientific minds that have ever lived have put more caveats and concerns in their published work than 98% of what is published today does. To get published, it seems you have to be 110% sure that your work is 120% amazing. I don't think this is a good state of affairs.

This is why different communities accept p-values at certain levels (typically <0.05). You have to live with some amount of uncertainty when the data generating mechanism is random... that is unfortunately the nature of the beast.

Hypothesis testing isn't the fundamental problem, it's how a given p-value is perceived. Getting a so-called significant result simply shouldn't be confused with conclusive evidence for the hypothesis.

Most studies use raw p-values. This, coupled with "exploratory analyses" which are not mentioned in the research, leads to a high false discovery rate.

Alternatives include using simulations to estimate false discovery rates (this can be done analytically for some problems). Bayesian frameworks can also be applied, depending on the amount and accuracy of prior knowledge of the system under study.

That said, I see nothing wrong with publishing results with nominal p-values, as long as the researchers indicate the weakness in their results. Meta-analyses can always come back later and use the results they publish.

What annoys me is seeing 50 or so statistical tests done, and then researchers stating they have found something when 1 of those tests shows a p-value of 0.05. Just using a Bonferroni correction, the simplest of all corrections, would demonstrate that findings like this are not significant.

Here's an article that makes the case that Bayes factors do a better job of summarizing the amount of evidence in favor of a conclusion: http://jnci.oxfordjournals.org/content/early/2010/12/03/jnci...

The problem isn't with the use of p-values, per se: the problem is that the human brain tends to distort probabilities on either end of the spectrum. People see that there's only a 5% chance that something could have happened by chance, and they put that in the "Not possible" bucket.

It's a fundamental weakness of the human brain. There's not a whole lot we can do about it.