A Statistical Test Gets Its Closeup

Great quote in the WSJ comment section:

http://blogs.wsj.com/numbersguy/a-statistical-test-gets-its-...

  The agency will bend their statistical significance
  requirements when hypothesizing that a treatment has
  risks, but almost never does when considering the 
  efficacy of a treatment.

That is the long and short of it. The FDA makes up the rules as they go along. As for the particular issue at stake, the Supreme Court's ruling in the Zicam case is horrifying:

http://online.wsj.com/article/SB1000142405274870371250457623...

It is indeed true that statistical significance is not the be-all and end-all. It is also true that you need some method of determining whether a result could be due to chance or not.

This is particularly true when you have trial lawyers and regulators who smell blood. Dow Corning was sued and had its reputation dragged through the mud because people thought breast implants were causing some kind of disease. Turns out that wasn't true at all:

http://query.nytimes.com/gst/fullpage.html?sec=health&re...

Now that statistical significance has been jettisoned as a defense, any group of anecdotal injuries of dubious provenance is in theory grounds for a class action lawsuit. And trial lawyers specialize in turning that particular kind of theory into practice. There's a reason that "mesothelioma" was the #1 Adword for a while -- ambulance chasers bid on it like crazy.

The Supreme Court has just made it a whole lot easier for them.

For the non-Bayesians, here's a paper that explains in detail one of the strongest criticisms of null hypothesis testing, by bayesian John Kruschke:

http://www.indiana.edu/~kruschke/articles/Kruschke2011PoPS.p...

To summarize, an experimenter's internal mental state (intentions) can affect the t statistic. That is: not changing the experiment at all, but just thinking something different -- gives a different level of significance.

In Kruschke's words: voo-doo.