Scientists use a value designated by p to describe the probability that a result arose by chance rather than design. In the social sciences, a p value of 0.01 - 0.05 is common, which means a result can be explained by chance with a 1% to 5% probability. As one moves through fields of greater rigor and seriousness, the p value required to declare a discovery becomes smaller. In experimental physics, 5σ (five sigma) has become the standard.
In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation). Experimental physics uses a one-tailed 5σ value, which is quite strict -- it has a numerical value of about 3×10-7. What this means is that an experimental result must be solid enough (and/or be repeated often enough) that the probability that it arose from chance is equal to or less than 5σ.
More here: http://blogs.scientificamerican.com/observations/2012/07/17/...
No. The Gaussian (normal distribution) is not involved. Sigma is just the symbol for standard deviation, that is, the square root of the variance. So, for random variable X where with expectation E[X] that exists and is finite (in practice a weak assumption), the standard deviation is square root of E[(X - E[X])^2].
So, the standard deviation is just a number, a measure of how 'spread out' the distribution is, and not an "area". Then 5σ is just 5 times the standard deviation.
[0] http://en.wikipedia.org/wiki/Chebyshev's_inequality#Sharpnes...
http://blogs.scientificamerican.com/observations/2012/07/17/...
Edit: oh, no, actually I think you're right :). I thought it was referring to the location of the test statistic in its null distribution. But it seems it's a scale for measuring p-values. This explains it clearly:
What does "five sigma" mean? It means that the results would occur by chance alone as rarely as a value sampled from a Gaussian distribution would be five standard deviations from the mean.
http://www.graphpad.com/www/data-analysis-resource-center/bl...
By the way, this same conversation took place after the LHC Higgs anouncement -- the same five-sigma standard for discovery, and the same detailed discussion of what that means.
> But it seems it's a scale for measuring p-values.
Yes, and that was a point I made in my reply to the OP -- that the context assumed an association with p-values, which in turn assume a normal distribution.
The only way that the Normal distribution comes into play is that physicists are measuring the smallness of their p-values by stating a number of standard deviations from the Normal mean that would have the same p-value.
I'm finding this discussion helpful by the way. I have worked in applied statistics but not in any fields which use this "sigma" scale or would talk about "sigma values".
No, not unless a p-value is expressed in terms of sigma as in this case and similar ones. In this case, and commonly in experimental physics, there's a relationship between n-sigma (usually 3σ or 5σ in different circumstances) and how a p-value is acquired from a sigma expression. The p-value is acquired from a sigma value like this:
http://i.imgur.com/pcjr6cN.gif
My point? In experimental physics there's a connection between (a) an expression including an integer and "sigma", (b) a resulting, widely quoted numerical value, and (c) the method for converting one to the other, using a Gaussian distribution as shown.
http://physicsbuzz.physicscentral.com/2012/07/does-5-sigma-d...
Quote: "But what does a 5-sigma result mean, and why do particle physicists use this as a benchmark for discoveries?
To answer these questions, we'll have to look at one of the statistician's oldest friends and C-student's worst enemies: the normal distribution or bell curve."
Couldn't have said it better myself.
> The only way that the Normal distribution comes into play is that physicists are measuring the smallness of their p-values by stating a number of standard deviations from the Normal mean that would have the same p-value.
Hmm. Yes, that's right. That's why I replied as I did in my original post.
In physics, a sigma value maps to a p-value, and that relationship is most often defined with respect to a normal distribution. Therefore, in most cases, to go from a sigma value to a p-value, one performs this integral:
http://i.imgur.com/YhC302b.gif
Specifically, the above definite integral, when performed with arguments of 5 and +oo, yields the often-quoted one-tailed p-value for "5 sigma", which we can get here as well:
https://www.wolframalpha.com/input/?i=5+sigma
It seems Wolfram Alpha makes the same default assumption I do: a normal distribution. If I weren't answering an inquiry from someone who wanted the clearest possible answer, I might have replied differently.
I'm absolutely correct in what I said.
But it appears that you are correct that that part of physics makes some Gaussian assumptions and, then, has some conventions based on those assumptions. In this case, apparently physics is not making its mathematical assumptions clear and explicit and is doing sloppy writing. For more detail, see my longer explanation in this thread
https://news.ycombinator.com/item?id=7421505
Maybe the situation is a little like getting from a little French restaurant the recipe for French salad dressing, sauce vinaigrette, making it at home, and concluding it tasted better in the little French restaurant. Hmm .... But the French restaurant did something not in the recipe -- took a large clove of garlic, peeled it, cut it in half, and wiped the salad bowl with the cut surface of the garlic! The recipe didn't mention that!
"In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation)."
He was just stating that the standard deviation is a general concept of spread, that any distribution has, and not just the normal one.
> "In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation)."
Not only is that not idiotic, that's the default definition in a statistical context (see below). Anyone can argue that σ is just another Greek letter with no special significance, but that require one to ignore the context in which the term is used.
Link: http://en.wikipedia.org/wiki/Standard_deviation
Quote: "In statistics and probability theory, the standard deviation (SD) (represented by the Greek letter sigma, σ) shows how much variation or dispersion from the average exists."
> He was just stating that the standard deviation is a general concept of spread, that any distribution has, and not just the normal one.
Again, this disregards context. When nontechnical people ask what the significance of 5σ is to scientific statistical analysis in physics (which is how this thread got started), there is precisely one answer.
In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation).
In statistics and probability theory, the standard deviation (SD) (represented by the Greek letter sigma, σ) shows how much variation or dispersion from the average exists.
I think you're missing the difference.
http://i.imgur.com/pcjr6cN.gif
Note that the unit normal distribution is the underlying context.
No, you're quite wrong about that. greycat is correct in his/her corrections of what you're saying. In statistics sigma is used to represent one of two things:
- a parameter of a probability distribution, typically one which influences the spread of the distribution
- a measure of dispersion in an actual data set, which may be an estimator of a parameter in a probability distribution
Neither of those things necessarily involve the Gaussian density.
In physics it seems that "sigma values" are used as a scale to measure p-values, so that instead of saying 0.0000003, they can just say 5σ. But the critical point here, which your comments seem to be missing, is that there is no distributional assumption being made; there is no implication that the Normal distribution describes any data-generating process, merely that the probability of an equal or more extreme value of a test statistic under some model, is the same as the probability of observing a value more than 5 standard deviations from the mean under a Gaussian model.
> No, you're quite wrong about that.
It is the default, actually. There are plenty of exceptions to the default, but it certainly is common in the context of experimental physics, the present context.
http://physicsbuzz.physicscentral.com/2012/07/does-5-sigma-d...
Quote: "But what does a 5-sigma result mean, and why do particle physicists use this as a benchmark for discoveries?
To answer these questions, we'll have to look at one of the statistician's oldest friends and C-student's worst enemies: the normal distribution or bell curve."
> ... your comments seem to be missing, is that there is no distributional assumption being made ...
Do read some experimental physics -- see what assumptions are made. Here is how a physicist maps a sigma value to a p-value:
http://i.imgur.com/pcjr6cN.gif
If this wasn't a discussion of the analysis of the outcome of a physics experiment, I would be more likely to accept these digressions.
lutusp, the way in which you are using the word "assumption" carries a very high risk that people will misunderstand you. The critical point here is that the physicists are nowhere using the Normal distribution as a modeling assumption. They are not suggesting that the Normal distribution is a reasonable model for any real data generating process in their problem domain. They are simply using it as a scale, like Celsius of Fahrenheit. There's a crucial philosophical distinction there that, even if you get, your readers will not.
> No.
Yes. The reason the various sigma values have the numerical values they have is because they represent integrals under the normal distribution, either one-tailed or two-tailed. Therefore in the present context sigma values represent definite integrals of the normal distribution.
> Then 5σ is just 5 times the standard deviation.
Yes, correct, except that the present conversation is about p-values and the meaning of sigma in this specific context. Five sigma has the value it does because it represents one minus the area between 0 and five sigma of the normal distribution, i.e. the one-tailed meaning of five sigma.
Link: http://blogs.scientificamerican.com/observations/2012/07/17/...
Quote: "A graph of the normal distribution, showing 3 standard deviations on either side of the mean µ. A five-sigma observation corresponds to data even further from the mean."
Quote: "For particle physics, the sigma used is the standard deviation arising from a normal distribution of data, familiar to us as a bell curve. In a perfect bell curve, 68% of the data is within one standard deviation of the mean, 95% is within two, and so on."
Picture http://blogs.scientificamerican.com/observations/files/2012/...
No. I defined sigma, that is, standard deviation, fully precisely and correctly. The normal distribution has nothing to do with that definition. And the standard deviation is a number, just a number, just as I defined it as
σ = E[(X - E[X])^2]^(1/2)
which clearly is just a number and not an area.
Or, for random variable X with cumulative distribution F_X, that is, for real number x,
P(X <= x) = F_X(x)
we have, with notation from D. Knuth's TeX, that
σ^2 = \int (x - E[X])^2 dF_X(x)
This integral need not be in the sense of Riemann (i.e., freshman calculus) because dF_X is a measure on the real line; so, the integral is in the sense of measure theory (see any of Rudin, Real and Complex Analysis; Royden, Real Analysis; Halmos, Measure Theory; Loève, Probability Theory).
Sigma is defined for any random variable X or its distribution provided that E[X] exists and is finite. Again, a "normal" or Gaussian assumption is not necessary. So, sigma is defined for discrete distributions, the uniform distribution, the Poisson distribution, the exponential distribution, etc.
For a random variable X, if don't know its distribution, then can't say what the numerical value of its standard distribution is.
Moreover if for some random variable X that has a standard deviation want to know, say, the probability
P( -5σ <= X <= 5σ)
then that is an area and need the distribution of X to find the numerical value.
All that is 100%, completely, totally, absolutely true. That's what σ or standard deviation is.
In particular, the statement
"In statistics, σ refers to an area under the normal distribution defined in terms of standard deviations (1σ = 1 standard deviation)."
is flatly false. The field of statistics has no such statement or convention.
Yes, if want to be really sloppy, make some assumptions not clearly stated, and have some conventions for identifying some things in special ways, e.g., that sigma is an area, then can do so. Maybe some parts of physics do this. I do remember when I was studying physics the prof handed out a little book on how errors were handled in physics. The book was a sloppy mess and one of the reasons I lost respect for accuracy and precision in physics and majored in math instead.
It is true that about 100 years ago some fields of study, especially parts of psychology and much of education, concluded that the Gaussian distribution was some universal law of data handed down by God. Well, God did no such thing. Still, some people in educational statistics believe that student test scores should have a Gaussian distribution and, if the scores do not have such a distribution, will, from many such scores, find the empirical distribution and, then, transform the scores so that the distribution is closely Gaussian.
Maybe physics drank that Kool Aid that all experimental errors of course, as given by God, have a Gaussian distribution, that is "a perfect bell curve", and, then, yes, can get
"68% of the data is within one standard deviation of the mean, 95% is within two, and so on.",
and that σ has a particular numerical value and regard standard deviation as an area. Yes, maybe this is physics but it is very sloppy thinking and not mathematics, probability, statistics, or anything from God.
Of course, even with a Gaussian assumption, standard deviation does not have a particular numerical value. Instead, for Gaussian random variable X with E[X} = 0 each of
P( -σ <= X <= σ)
P( -2σ <= X <= 2σ)
P( -5σ <= X <= 5σ)
has a particular numerical value.
Sorry 'bout that.
That's very interesting!
What are some practical or pragmatic advantages of your way of thinking compared to the standard (sloppy) way of thinking? I don't mean to put you on the spot; I genuinely would love to know.
For example, what are some statistical problems which the standard (sloppy) mental model would have a tough time solving, but which your mental model would be able to yield tools for?
But the physicists who apply this method to their LHC data, and those who apply it to the newer finding of gravitational waves, know exactly what they're doing. The data being analyzed are entirely appropriate to this method, and the conclusions being drawn are sound.
So, suppose we assume a null hypothesis that all 35 student scores are independent random variables with the same distribution, that is, (1) Mary and Bob are equally good as teachers and (2) their students are equally well qualified. If the students had been appropriately randomly assigned to Mary and Bob, then maybe we can believe (2) so that only (1) is in question. So, we are going to test (1), that is, that Mary and Bob are equally good as teachers.
This hypothesis, that is, that Mary and Bob are equally good is called a null hypothesis since it assumes that there is no effect between Mary's class and Bob's class (even though Mary is teaching more students than Bob).
So, here is how we do our test: We throw all 35 scores into a pot, stir the pot energetically, pull out 20 in one bowl for Mary and put the other 15 scores into a bowl for Bob. Then we average the scores in each bowl and take the difference in the averages, say, Mary's average minus Bob's average. Then we repeat this many times -- for this might want to use a computer with a good random number generator. This process is sometimes called 'resampling'. It's also possible to argue that what is going on is a finite group of measure preserving transformations that, thus, can yield what we are doing more intuitively. Physicists like symmetries that result in conservation laws, but here we have symmetries resulting in an hypothesis tests.
So, we get the empirical distribution of the differences in the averages.
Then we look at the difference in the actual averages, that is, from the actual students of Mary and Bob. Call this difference X. Now we see where X is in the empirical distribution of differences we found. If X is out in the tails with probability, say, 1%, then either (A) Mary and Bob are equally good as teachers, that is, we accept the null hypothesis, and we have observed something that should happen 1% of the time or less or (B) Mary and Bob are not equally good as teachers and we reject the null hypothesis and conclude that there is a difference in teaching between Mary and Bob.
If the 1% is too small to believe in, then we accept (B) and pop a Champaign cork for the better teacher.
Here we made no assumptions at all about the probability distributions of the scores. So, our hypothesis test does not assume a distribution and is distribution free or, as is sometimes said, non-parametric (because we did not assume a distribution, say, Gaussian with parameters, e.g., mean and variance).
Look, Ma, no standard deviations!
The 1% is he significance level of our hypothesis test and is the probability of rejecting the null hypothesis when it is true and, thus, is the probability of Type I error.
Oh, consider a large server farm or network. Suppose we identify 10,000 systems we want to monitor for health and wellness and detect problems never seen before. Suppose from one of the 10,000 systems, we consider one. Suppose from this system we receive data 100 times a second on each of 12 numerical variables.
Suppose the server farm is supposed to be fairly stable and we collect such data for, say, 3 months. Call this history data. Maybe the machine learning people would call this training data. Whatever.
Now we can construct a 12 dimensional, distribution-free hypothesis test where the null hypothesis is that the system is healthy and also select our false alarm rate (probability of Type I error) in small steps over a wide range. So, we have a multi-dimensional, distribution-free hypothesis test. Such are rare, but, really now we have a large class of them. Yes, we use a group of measure preserving transformations.
As I recall, back in 1999 there was a paper on such things in Information Sciences. I wouldn't call that paper machine learning, but maybe some people would. Some of what is interesting in the paper is how the heck to know and adjust the false alarm rate, that is, the probability of Type I error.
Again, look, Ma, no standard deviations or Gaussian assumptions.
You're once again overlooking the context. In a physics context, 5σ has an oft-quoted numerical value that is acquired this way:
http://i.imgur.com/pcjr6cN.gif
Good luck acquiring the universally accepted numerical value without applying a Gaussian distribution as shown.
Remember that this thread began with someone nontechnical asking what the significance of 5σ was to the evaluation of a physics experiment, for which I provided an uncontroversial explanation in that context.
Yes, and if I graph a function in terms of X and Y, where X is a function's argument and Y is the value returned by a particular function, one can argue that X and Y are just placeholders for numbers without any intrinsic meaning by themselves.
If I then say that Y represents the sine of X, surely someone will say, as you have said, "No, not at all, X is just a placeholder for a number, it's not what you say. And Y is just a placeholder for a number, it's not tied to any particular function."
In point of fact, the standard deviation is more than a particular number, it's an idea, and its statistical purpose is met when it's associated with a context in which that idea is expressed. That context is the normal distribution. In the present context, a particular sigma value refers to a specific area under a normal distribution, and in turn, to the probability that a particular result might have arisen by chance.
Without reference to a normal distribution, a standard deviation (a sigma) loses its conventional meaning. Variances are acquired by statistical tests of data sets, standard deviations (sigmas) are acquired from variances, and conclusions are drawn from sigmas only to the degree that they are applied to normal distributions.
Link: http://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule
Quote: "In statistics, the 68–95–99.7 rule, also known as the three-sigma rule or empirical rule, states that nearly all values lie within three standard deviations of the mean in a normal distribution. About 68.27% of the values lie within one standard deviation of the mean. Similarly, about 95.45% of the values lie within two standard deviations of the mean. Nearly all (99.73%) of the values lie within three standard deviations of the mean."
According to your thesis, the above claim is obvious nonsense, because in point of fact, sigma has no association with the normal distribution. But you know what? Wikipedia can be edited by anyone, and you can correct this egregious error today, if you like. Correct this widespread erroneous thinking -- fix these errors, everywhere you find them.
Let's perform a little test. Let's submit "5 sigma" to Wolfram Alpha and see whether it makes the same mistake you say I have been making, and that the above linked article is making:
------------------------------------------------------
Link: https://www.wolframalpha.com/input/?i=5+sigma
Quote:
Input Interpretation: 5 sigma (standard deviations)
z-score : 5
Probabilities:
zɝ (left-tailed p-value) | 1-2.867×10^-7
zɱ (right-tailed p-value) | 2.867×10^-7
abs(z)ɱ (two-tailed p-value) | 5.733×10^-7
abs(z)ɝ (confidence level) | 1-5.733×10^-7
Associated two-sided confidence level: 100-5.733×10^-5%
So it seems that, without prompting, Wolfram Alpha draws the same conclusion that Wikipedia does, and every other online reference does when confronted by terms such as "standard deviation" or "sigma" -- that, without providing a specific context, the default context is that sigmas refer to positions on a standard distribution and have statistical meanings associated with that default assumption.
> Of course, even with a Gaussian assumption, standard deviation does not have a particular numerical value.
When a scientist uses the term "5 sigma", she is in most cases using a context of a normalized normal distribution, one that follows the 68-95-99.7 rule described above. Therefore yes, 5 sigma does lead to a a particular value, by being applied to a definite integral of a normalized normal distribution that uses it as an argument. By this reasoning, 5 sigma means:
(Sage notation) integrate(e^(-x^2/2),x,a,b) / sqrt(2 * pi)
Picture: http://i.imgur.com/YhC302b.gif
If I try to compute the value used in physics, the result for 5 sigma, I can use this:
(Sage notation) N(integrate(e^(-x^2/2),x,5,infinity) / sqrt(2 * pi))
With this result: 2.86651571870318e-7
Which is correct, and is the expected one-tailed value for 5 sigma, even though according to your argument, 5 sigma has no connection to normal distributions.
> Sorry 'bout that.
What, for your specious argument? No problem, it comes with the territory. But surely you realize I have more than answered your objection.
No. Here is an important use of standard deviation not related at all to the Gaussian distribution: The set of all real valued random variables that have a standard deviation and one that is finite form a Hilbert space. The crucial part of the argument is completeness. Of course we like to use Hilbert space for projections and converging sequences, so it's super nice that those random variables form a Hilbert space.
> No. Here is an important use of standard deviation not related at all to the Gaussian distribution ...
Before you troll through the world's imaginary problems again, I am going to ask you one more time to remember how this conversation got started (the context), and why I answered as I did.
Someone nontechnical asked what the significance of 5σ was in the context of a physics experiment. I replied by saying that 5σ was mapped to a p-value this way:
http://i.imgur.com/pcjr6cN.gif
And the p-value was the point, leading to a discussion of the high level of discipline in experimental physics compared, say, to the social sciences, which accept p-values of .01-.05, values I included in my original reply for comparison.
The context is the normal distribution. Wake up and smell the Cappuccino.
And, if some field makes a normal, or mean 0, variance 1 normal, assumption, then they should say so and, hopefully, justify that assumption.
Your examples from Wolfram and Wikipedia show that a lot of people have guzzled that old Kool Aid on the normal distribution. A few years ago I had a date with a high school teacher, and she assumed that test scores are normal. My father had an MS in education and my brother was a ugrad psych major, so, when my brother's psych material got to statistics, Dad taught him about the normal distribution. It appears that around 1900 and then for 80 years or so, and still in some parts of some fields, a normal assumption, without mention or justification, was common and remains in what you call the "context". "Context" or not, common or not, popular or not, it's still an assumption, needs mention and justification, and usually is not at all well justified or justifiable.
Sure, with their normal assumption and, say, five sigma, experimental physicists get particular probabilities of Type I error, so called p values. No question here.
The issue is the normal assumption.
There are places where a normal assumption is quite solid. The most common justification is from the central limit theorem. So, take random variables X(1), X(2), ... and assume that they are independent and all have the same distribution with finite standard deviation. Then the central limit theorem says essentially that as positive integer n grows to infinity the distribution of the sum
X(1) + ... + X(n)
will converge to normal (really should divide the sum by square root of n).
So, where might we get some such
X(1) + ... + X(n)?
Sure, from Brownian motion where there are many little bumps and the bumps come close enough to satisfying the hypothesis. So can use this also in thermodynamics.
So, there are places where a normal assumption is justified.
But just making a normal assumption for essentially all experimental errors as needed for the accepted five sigma criterion is close to a weird religion and not flattering to a modern science.
What physics is doing with their five sigma criterion is a statistical hypothesis test. My examples of distribution free hypothesis tests via resampling is a better justified and more conservative and robust way to do an hypothesis test. Physicists might consider using such.
Such distribution-free statistical methods form a large field.
My example of the role of Hilbert space is quite practical.
If you want to explain to the common man in the street why physics likes their five sigma criterion, then, sure, you need the normal assumption. Then you should mention that there physics is making a normal assumption.
There you might not confuse the common man in the street with a claim that in making this normal assumption physics is close to drinking some swill of boiled tails of rats and bats.
But with the now high interest in computer science of big data, machine learning, etc. I was assuming that the HN audience could and should hear the real stuff -- that in what physics is doing, there's a normal (Gaussian) assumption in there.
That's not what p value means. The p value is the probability that a high-variance random effect (centered around an average behavior of "nothing interesting") would yield a result as extreme as the observation, assuming that random distribution. You need Bayes theorem and highly subjective assumptons if you want to derive a posterior probability that an observed result was drawn by chance from a sample with a boring/interesting mean.
> That's not what p value means.
Yes, I know. Verbal shorthand and some associated risk of being misinterpreted.
"Particle physics uses a standard of "5 sigma" for the declaration of a discovery. At five-sigma there is only one chance in nearly two million that a random fluctuation would yield the result. This level of certainty prompted the announcement that a particle consistent with the Higgs boson has been discovered in two independent experiments at CERN."
"When the uncertainty represents the standard error of the measurement, then about 68.2% of the time, the true value of the measured quantity falls within the stated uncertainty range. For example, it is likely that for 31.8% of the atomic mass values given on the list of elements by atomic mass, the true value lies outside of the stated range. If the width of the interval is doubled, then probably only 4.6% of the true values lie outside the doubled interval, and if the width is tripled, probably only 0.3% lie outside. These values follow from the properties of the normal distribution, and they apply only if the measurement process produces normally distributed errors. In that case, the quoted standard errors are easily converted to 68.3% ("one sigma"), 95.4% ("two sigma"), or 99.7% ("three sigma") confidence intervals."