Hacker News Reader

To be explicit, since this comment is still top voted: This comment was made by someone who only watched the first of 2 presentations.

The second presentation showed a 5 sigma result.

I'm Incredibly disappointed by the trivial inaccuracy of comments on hacker news lately, and that corrections never get upvoted quickly enough to prevent the spread of misinformation.

> which would mean a 1 in 1.74 million chance that the results occurred by chance rather than being a signal

No!! It's the chance that randomness could produce a result that large. This distinction sounds pedantic, but misunderstanding this is widespread and leads to fallacies committed frequently by very smart people who ought to know better in many fields.

Aren't those the same thing? How are they different?

I suspect Almaviva is talking about the biases inherent to publication / talking about results.

Consider, by analogy, the event "rolling a six sided die and getting a 6 and announcing that fact to the world".

"What is the probability that random events could produce a result that large?": one in six, per die roll. The question excludes the whole "announce it to the world" filter.

"What is the probability that these results [getting a six and announcing it] occurred by chance, rather than being a signal?": We have no idea. If the person announced "I'm going to roll one die and announce the results, regardless of the outcome", then it's one in six. If they kept rolling dice until they got a six, then the probability is 1. If they rolled 3 dice, then the probability is 91/216.

The point is that the scientific method has all sorts of biases (publication bias, confirmation bias, etc.) and p-values are rarely "probability that the result is wrong".

Standard null hypothesis testing -- including what they're talking about here -- focuses on

P(results | random noise is active)

You're talking about

P(the thing we care about | results)

The former is more like a simple sanity check. If random noise could have produced what you see, you shouldn't take the results too seriously.

The former tells you a little bit about the latter -- which is good, because the latter is what we actually care about. But you can't explicitly compute the latter without making much stronger assumptions like priors and the like. That's why this last step of reasoning is often performed qualitatively.

A good stats book to read about all this is Larry Wasserman's http://www.stat.cmu.edu/~larry/all-of-statistics/

Imagine testing 1000 potential cancer drugs. Only 100 of them actually work, but you don't know that; you have to do the trial. So you get out some petri dishes and start testing. You look for a 5% chance of false positive (p < 0.05), which is the statistical significance level usually used in medical trials.

Of the 100 real drugs, you detect all of them. Of the 900 fake drugs, 5% of them falsely appear to work. So you have 145 drugs you think work, only two thirds of which actually work. The chance of any individual drug having obtained its positive results by chance is 31%, not 5%.

I have a guide to this here, since it's so common:

http://www.refsmmat.com/statistics/#an-introduction-to-data-...

(Side note: the average medical trial would only detect 50 of the working drugs, not all 100, so the real situation is even worse.)

> The chance of any individual drug having obtained its positive results by chance is 31%

And that's different from the chance that randomness could have produced the effect? How?

Sorry, I wasn't very clear. Usually people quote the p value as the chance the result is a fluke; the p value in CERN's case is p = 1/1,740,000. But that's the chance that the effect would be produced if the Higgs did not exist, which is different.

By analogy in the medical case, p = 0.05. The incorrect interpretation is that this means only 5% of drugs with statistically significant benefits actually achieved these benefits through luck; rather, the right interpretation is that 5% of the nonfunctional drugs somehow appeared to work.

You could also imagine testing 200,000,000 hypotheses which were all completely false. Even if you used CERN's level of statistical significance, you'd still quite likely find one hypothesis which appears to be true, simply by chance. The chance of that hypothesis being false is 100%, despite the significance level of 1 in 1,740,000.

So yes, 31% is exactly the chance that randomness produced the effect in the trial. But people will try to tell you that it's actually 5%, and they're wrong.

This thread started with the claim that "the chance that the results occurred by chance" was different from "the chance that randomness could produce [the result]".

But you're saying that in your example both are 31%. So again, I ask, are we talking about two separate things? And if so, can you give an example where the two things have different values?

In my medical example, "the chance that the results occurred by chance" is 31%. "The chance that randomness could produce [the result]" was only 5%.

For CERN, the chance that randomness could produce this result is 1 in 1.74 million; the chance that the results occurred by chance is larger, but not computable with the information we have,

The guide I linked to above gives a much better explanation than this. I rushed my first post here, and I think I was unclear.

I swear I'm not trying to give you a hard time, but I don't see how these two things you said could both be true:

> "The chance that randomness could produce [the result]" was only 5%.

> 31% is exactly the chance that randomness produced the effect in the trial.

Whoops, poor wording on my part.

Imagine flipping a perfectly fair coin 100 times. You'd expect to see 50 heads, but you don't always -- it's just an average. Suppose you see 75 heads. What is the chance that you'd see 75 heads with a fair coin? Very very small. The chance that randomness could produce such a result is small.

Now, imagine you test 100 perfectly fair coins. A few of them give more than 75 heads, just by luck. You conclude they're unfair, since the result is unlikely otherwise. The chance that randomness produced the effects you saw is actually 100%, because all the coins are fair.

There's a difference between the question "How likely is this outcome to happen if the coin is fair?" and "Given that this outcome happened, how likely is it that the coin is fair?" Statistical significance addresses the first question, not the second.

Thanks; I feel like I learned something. Your infinite patience was appreciated.

A couple of extreme examples:

1. I'm rolling two dice to try and get the highest total. I get two sixes. What is the chance randomness produces this? 1/36, about 0.027. This more than a two sigma result. What is the chance that this is caused by random chance? 2.7%? Nope, 100%.

2. I study the same thing in a million situations in parallel. I take the most extreme result and find that random chance can produce this one time in 1.7 million. It's a five sigma result! What are the chances this result is caused by random chance?

The 5σ convention is more about announcement than about discovery. No one can really say when a discovery is made. As more information comes in, the image can remain fuzzy or can get clearer and clearer. In the latter case, 5σ is a convention as a threshold for when you are entitled to announce a discovery without being blamed for jumping the gun. They'll collect data for many more years before passing the threshold of Nobel-worthy discovery.

Atlas is still presenting. So let's wait and see what they came up with. 4.9σ was just from CMS.

Edit: It is worth noting that 125GeV fits well within Fermilab's recent announcement: http://news.discovery.com/space/tevatron-data-detects-higgs-...

I think these physicists have been watching too many Steve Jobs presentations. "4.9σ... Oh, but one more thing!" ;)

In all seriousness, does anyone know how common it is to do this kind of gradual reveal during scientific presentations vs. stating your basic final results in the introduction? It's kind of fun, but you could cut the tension in that room with a knife!

The way they handled it was perfect. For over 20 minutes each, the presenters for CMS and Atlas explained their measurement capabilities, methods, recent improvements, and recorded measurements before stating their conclusion. They made sure to cover their bases and justify the conclusion before making a big announcement. I'm pretty sure the abstract for the papers will state the conclusion early on but the initial announcement for the greatest achievement in Particle Physics shouldn't be prefaced with "We found it!" just for the convenience of the impatient.

The bravado comes from an ego and pride in ones self worth, not from Jobs' keynotes.

She is working the crowd, so to speak. Here is the final CERN press release, which claims a 5 sigma result:

http://press.web.cern.ch/press/PressReleases/Releases2012/PR...

5σ is not a rule. It's just a measurement of the probability that the result you observe is an accident. It means that it's very unlikely that it was just pure luck.

By the way, σ measurements are also used in many engineering jobs, and quality systems measurements. The kind of industry "goal" is to get 6σ (let's say when you manufacture consumer goods in large quantities, like millions every month). But for some other industries, 9σ is the benchmark - and from what I know, air companies have quality systems up to 9σ to ensure the lowest risk of something wrong happening.

How can they possibly have up to 9 sigma? 6 sigma is a failure rate about 1 in 10 billion, whereas 9 sigma would be 1 in 10^19 or so! That's oddly similar to the number of grains of sand on the earth![1]

[1] http://www.hawaii.edu/suremath/jsand.html

The math will happily pop out absurd numbers. Remember that all stats carry assumptions with them, such as "this is a Gaussian process from negative to positive infinity" and "everything is absolutely perfectly independent" and other things that break down if your push them hard enough. What that is telling you is just how hard you're pushing the math rather than the real odds. I'm not 1 in 10^19 sure of anything.

Yeah, I doubt most pilots have failure rates anywhere near or above 4 sigma.

That's not about pilots. It's about equipment failure rates.

Where did you get your failure rate at 6 sigma to be 1 in 10 billion ??

6 Sigma is 3.4 defects per million opportunities. http://en.wikipedia.org/wiki/Six_Sigma

>IANAPhysicist, but I'd be interested to know how strict the 5-sigma discovery rule is considered - for example, could they still get a Nobel prize for a 4.9σ announcement?

It's very much a HEP thing, and done that way because HEP is pretty much all statistical analysis these days. Other fields wouldn't treat the sigmas as the most important thing, and I've heard mutterings that it's not really the most accurate approach - but it is objective and easy to apply.

You should have waited to watch the ATLAS woman's presentation afterward. During hers, they concluded with a solid 5 sigma.

5σ seems like an unnecessarily high standard to this non-physicist; what's the rationale for that? At 5σ we could publish 1,000 major discoveries a year and have E[false discovery accepted] = 1,740 years.

I think it's mainly so they can be as condescending and smug as possible at interdisciplinary conferences.

Actually, it's because the rest of you "scientists" end up publishing bullshit results that you got by chance.

I'm kind of joking - most other scientists don't collect enough data to have to worry about 1 in 10,000 events happening by chance. In medicine, though, I'm not joking at all - those guys publish absolute statistical garbage all the time, I hesitate to even consider it a science because the data dredging is so bad. I can "prove" just about anything if the publication standard is 95% significance...

The two main rationales are that systematic uncertainties are historically under-estimated and that we are looking in many channels (more than 1000), so it would not be too hard to find a 3 or 4 sigma anomaly. The second part is the so-called "Look Elsewhere Effect." If you hit 5 sigma, you are fairly safe from either of these effects ruining your "discovery."

> The second part is the so-called "Look Elsewhere Effect."

Also more generally known as the "multiple testing" problem, fwiw (not sure why it has a different name in physics, unless I'm missing a subtlety).

It's a major problem in "big data" also, where people just data-dredge thousands of possible parameter choices and pairwise correlations, and then report the p<0.01 results that came up, even though you'd expect several false positives just by chance with that methodology.

I searched but didn't find an explanation of this 5 sigma. Any reference to share?

5σ = 1 / 1,744,278 chance this was just luck.

http://en.wikipedia.org/wiki/Standard_deviation

The question this actually answers is: Assuming it was luck (ie there's no Higgs), how likely is it to get the measured result (or one even more extreme)?

Exactly. P(results|luck) and P(luck|results) are two very different things. We don't know the probability that the results are due to "just luck" (whatever that means).

I think it's quite dramatic how many very intelligent people (up to writers in major publications) are uneducated about this distinction, and how dramatically wrong decisions can be if this is misunderstood. If we could teach this, as well as correlation vs causation, I'm convinced the intelligent public would make much better decisions about many things, medicine and nutrition to name a couple.

I know standard deviation but I didn't know scientists reported their results talking about x sigmas. Thanks for clearing that up.