Hacker News Reader

Comment by tomjen3 | Hacker News Reader

Buy that same table at 3 sigma there is still 99% chance that they have found the damn thing.

Isn't it usually accepted that it is statistically significant when p goes above 0.95? If so, why aren't they happy with a 20 times better result?

View on HN

Particle physicists generate data by the petabyte. If they used 3 sigma confidence they'd confidently declare all sorts of false things. We know this, because if you pay attention to particle physics you hear about all kinds of 3 sigmas that turn out to be spurious. (About every six months to a year or so, I'd say.) 5 is just where they start to feel comfortable, they'll take more if they can get it.

http://www.nature.com/news/2011/110119/full/469282a.html

Your answer is good and sufficient. For people who aren't yet getting an intuitive feel: if 99% confidence means you're right 99% of the time (it doesn't quite mean that but lets go with it), then you'll still be wrong 1% of the time. If 3000 studies are published each year with that cutoff for significance, you should expect about 30 spurious results accepted as true each year. Ouch.

That reasoning neglects the bias towards papers with interesting results, which leads to a much higher rate of false reports in hot journals.

In human genetics, studies will meta-analyze each of the ~million SNPs tested across each study, rather than just picking the "candidate" SNPs that came to attention because of their association signal in any particular study. So the file drawer problem is still relevant, but the risk is greatly mitigated.

Because 3 sigma doesn't mean 99% chance they've found it; it means that, if it doesn't exist, there's a 1% chance of obtaining the results they did. Common statistical misconception.

Consider a situation where they test 1000 hypotheses, only 100 of which are actually true. Of the 900 false hypotheses, 9 will, by chance, achieve a 3 sigma "99% chance of being right" result. And hence something like 8.3% of the hypotheses tested as "true" will be false positives -- not 1%.

Particle physicists usually require 5 sigma for a discovery. They've apparently had plenty of 3 sigma results that didn't hold up, so they have high standards. 3 sigma to them now is merely the point where people start to get interested in your result.

Actually it's usually 6.

Are you satisfied with a 1 in 20 chance of complete BS for the foundations of physics?

Biology uses .95 because it's virtually impossible to get better results than that. But it also means that 1 out of every 20 biology papers are completely wrong. And then you have papers based on those....

Physics wants a much greater degree of certainty, and since they have the ability to get it they insist on it.

This http://xkcd.com/882/ sums it up pretty well.

You have to make sure to have results that are far far better than the number of experiments you are running. Otherwise you are virtually guaranteed to find some result that will seem right, but isn't.

(For example a 1 in a million occurrence (per person per day) [would] happen about 7000 times every day.)

That "completely wrong" is a bit harsh on biologists.

Firstly, it is a lot harder to repeat experiments in biology than in particle physics, as (as far as we know), all electrons are the same.

Secondly, biologists will, in general, not make bold claims. A paper "a possible link between X and Z" that works with p>0.95 and states that further research is needed is not a lie; the popular press makes it a lie by changing it to "OMG: X CAUSES Y".

I know it's a lot harder to repeat experiments in biology than in particle physics - I said that in my post.

Just because someone uses weasel words ("possible") doesn't change the end result: It's a wrong result.

I'm not blaming them - I understand better results are not possible. But it doesn't change the fact that a tremendous number of results are wrong.

It doesn't help that they often search for very subtle results. "It helps, but only a little." It also doesn't help that everyone responds differently to things. It makes the research very hard.

Anyway, I was just explaining why p95 is not accepted anywhere else except biology - biology just doesn't have any other choice. They don't prefer such low results.

Different fields have different acceptable levels of significance. Social science and engineering fields generally except significance at 95% or 99% levels , physics has higher standards.

Because of this: http://xkcd.com/882/

That's not how statistics works. Your statement is the converse of happened, not the contraposive.

Reply on news.ycombinator.com