I'm a high school science teacher. Could you point me to the typical methods you use to calculate the level of confidence of results at CERN?
This is possibly the most highly-cited statistics paper in our field: https://arxiv.org/abs/1007.1727 It provides asymptotic formulae that allow us to estimate these p-values without having to simulate our experiment tens of thousands of times. But it also gives a nice overview of the different test statistics one might use.
"LHC Statistics for Pedestrians" is short and informative, but a bit haphazard and probably assumes you're familiar with a lot of jargon: https://cds.cern.ch/record/1099994/files/p205.pdf
"Practical Statistics for the LHC" is a much more complete and thorough introduction, but it's quite long and technical: https://arxiv.org/abs/1503.07622
"Statistics for searches at the LHC": https://arxiv.org/abs/1307.2487 Similar to the previous one, but maybe more words and less math. Perhaps one of these will resonate with you more than the others!
Lastly, here is a public note that was created jointly by the ATLAS and CMS collaborations prior to the Higgs boson discovery. Basically, we got together and came up with a consistent way to present new results from the LHC, and this note documents those protocols: https://cds.cern.ch/record/1379837
I don't see the purpose of question number 1, it seems totally spurious. All you need to care about is how well the various models explain the data.
EDIT:
For example, with the famous "sun's gravity bending starlight" example they compared the observations with predictions due to Newtonian gravity x with predictions due to Relativity 2x, there was no need to check for zero deflection since there was no theory predicting that. However, if some value (including zero) inconsistent with both was observed a new theory would need to be developed. Then that theory would need to be tested on some other phenomenon.
You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis. Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true.
For instance, in your example, if they had observed 1.8x, but the uncertainty on that were 0.8x, no discovery would be claimed despite the relativistic model explaining the data better than the non-relativistic model.
The data is consistent with the predictions of both models in this case. If we want to distinguish between them we need to get more/cleaner data or different data to compare to different predictions. What is wrong with that?
>"You don't just try to find the model that best fits the data and claim a discovery if this isn't your null hypothesis."
This is not what I am suggesting to do, see above.
>"Rather, the data needs to be far enough away from that predicted by your null hypothesis that it is highly improbable that you would find data at least that far from the prediction if the null were true."
What purpose does this serve?
EDIT:
Here is a venn diagram of what I mean, with four possibilities: https://image.ibb.co/c432OT/venn_Science.png
You are saying for some reason it is important check for !B which equals regions A - (A or B) + !(A or B). Why?
Why not just check which corresponds to the results? Is it model A only, model B only, model A and model B, or neither model A nor model B?
In principle that's fair enough, but in practice, the Standard Model has already passed lots of tests -- it fits the results of many, many experiments -- so we already have a high confidence that it's correct (or rather, a very close approximation to the truth for all the experiments we're been able to try). So brand new model B does not start on an equal footing. The null hypothesis is "model A is correct".
If there's some new experimental result that does not fit with the Standard Model -- and this usually not a clear-cut thing, but a statistical estimate of likelihood -- but does fit model B, that's a useful new result. If the error bars are such that it fits both models, you stick with the tried and tested model A, unless B has some compelling new advantage (like being much simpler than the Standard Model, say) and it also fits all other existing experimental data.
Also model A may be something like "Standard Model w/o sterile neutrinos", while model B is "Standard model w/ sterile neutrinos". My understanding is these will be supported by all the same previous evidence.
And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose?
In that specific case, no, a number of existing experiments appeared to confirm the "w/o sterile neutrinos" model, so it was assumed that was correct. Just one older experiment fit the "w/ sterile neutrinos" model; that has only recently been reproduced successfully, so now that model is back on the table. But it's still possible that it's just a measurement error.
And anyway, if you want to include prior info there is a principled method of doing so: Bayes' Rule. The method described here is indirect, obfuscated, qualitative, and open to all sorts of bias. In general it seems inferior in every which way, why has Bayes' Rule been rejected for this purpose?
Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics?
I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what? I don't think you can easily say "the Standard Model is incorrect" -- we know it's incomplete, hence incorrect, but it also fits experimental data incredibly well, so it must be a close approximation to some level of reality at the very least.
(edit to clarify: what I mean is, I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)
I was talking about all the other types of observations used to support the standard model. Afaik the existence of sterile neutrinos would not change the predictions.
>"Good question, I don't know! I wonder if some of the LHC people commenting here can talk about the reason for sticking with Frequentist over Bayesian statistics?"
It isn't really a matter of frequentist vs bayesian statistics. Frequentist stats looks like what you see in Figure 1 here (my venn diagram is just a simpler version of it): http://rsta.royalsocietypublishing.org/content/231/694-706/2...
However, it is put forward that prior information should be involved in interpreting the results of the significance test. This idea has been rejected by all the giants and founders of modern stats (Fisher, Neyman, Pearson, etc). Its just some ad hoc thing come up with by stats 101 authors.
>"I suspect this gets into some pretty deep issues around the philosophy of science. Are we looking for the "correct" theory, or just successively better approximations to reality, or what?"
We're supposed to be comparing the predictions derived from our theories/models to data collected after the prediction was made and use these predictions to come up with useful tech and perform amazing feats.
>"I'm not sure how you set a prior confidence on "is the Standard Model correct?" in a Bayesian sense, since we already know it's partially correct and partially incorrect.)"
The current solution seems to be performing an elaborate ritual that obscures the true method of using collective opinion.
For instance, the Higgs is part of the Standard Model but for the Higgs discovery the null hypothesis was that the Higgs does not exist.
>"Assuming the background estimation is correct, the p-value will only converge to zero if a true signal exists."
Aren't you assuming the background model is correct and a true signal exists?
>"Your null is specifically "no signal exists"."
No, it is never this. It is an entire model, one assumption of which is "no signal exists". The p-value doesn't care which assumption is wrong, it is only wishful thinking that leads to people focusing on that one.
>"We don't know that the null is flawed to begin with."
Sorry, I don't believe this. Give a concrete example. Do you believe the standard model is 100% correct? If not, then any background derived from it must be assumed to be flawed to begin with.
>"Things like malfunctioning equipment go into the uncertainties if they occur in a way that the researchers have considered (which often means uncertainties are set conservatively if the equipment behaviour is poorly understood). If they occur in a way that no one considered, there's no statistical trickery that's every going to compensate for that."
Correct, there is no statistical trickery to compensate for this. It is a scientific problem, not statistical.
Did you check out that Meehl paper I linked elsewhere?[1] If you take a prediction of a model and test that, messing up the experiment means you get results that diverge from your prediction. If you reverse the logic of science so that you test the "opposite" of your prediction, messing up the experiment yields results that seem to support your model. This is why the prediction of the theory needs to be set as the "hypothesis to be nullified".
[1] https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte...
I know you're feeling very right and vindictive but damn if this isn't tiring to read. Make sure you know what you're right about first before assuming the other thing is wrong just because it matches a few keywords that you've read are bad. Because assuming good faith means that you allow for the possibility they might not be doing it wrong. You still get to make your point but it saves your face a lot.
I'd suggest, go outside, attack a windmill for a few hours, think about what you're actually trying to say, and if it's not "hey I read this article on lesswrong and it says everybody is doin it wrong", you'll probably find you can make your point much clearer.
You seem to have more than sufficient knowledge about statistics to think for yourself on this matter.
Quote this. Where? I am sure by "corrected" you mean "someone claimed something I believe to be wrong, so I continued to point out why." Amazingly, this looks like yet more strawman arguing...
1. If you're using a data driven background, you're background model isn't the Standard Model but that there isn't something special about your signal region.
2. If you are using the SM as your background, then yes you do believe it to be 100% true, since what you are looking for is evidence that there exists new physics not described by the SM.
In other words, the theory under test is the SM, and it can pass very easily. If it fails despite this, that's evidence that it's incomplete. If it passes, the limit setting stage tests whatever new physics we were looking for, and gives an upper bound on how sure we are that it does not exist.
Sure, it means you assume that during the signal detection everything was working exactly as it was during background data collection. There are going to be other implementation specific assumptions like what thresholds to use regarding environment noise triggers, accounting for sensor drift, etc. These data-driven models are approximations, I doubt the people actually coming up with them really believe they are 100% true. If you give a real life example I will point out exactly where there are issues.
> "If you are using the SM as your background, then yes you do believe it to be 100% true"
I don't know what this is supposed to mean. Of course you are assuming your background is 100% true. Do particle physicists actually believe this though? Not from my reading:
"So although the Standard Model accurately describes the phenomena within its domain, it is still incomplete." https://home.cern/about/physics/standard-model
As for the SM, we are looking for those places where it is incomplete, and to do this we must assume it is complete and look for strong evidence that we are wrong. To date, it doesn't seem like we see though, as far as collider physics is concerned.
Yes, I know. This isn't at all a positive thing as you seem to think...
I've been lately seeing that this practice is infecting particle physics, which is not a good sign. I do not think they are overwhelmed yet though. Here is how it has gone in every field so far (education research, psychology, medicine, etc):
Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct.
The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?
Not just any old favorite theory. For instance, an excess at a mass of 125 GeV doesn't provide any evidence for a new particle at 2 TeV. It does however provide evidence for a new particle at 125 GeV.
> Eventually the field will stop caring about all actual quantitative predictions (some barely funded crackpots on the edges will publish in ignored journals) and researchers will only test a "null hypothesis" that everyone knows is wrong (eg, in the physics case a world where the standard model is assumed true but has no higgs boson). Then upon rejecting the (known to be false) null hypothesis, they will conclude some theory only capable of vague/malleable predictions is correct.
If that were the case, we would have made a lot more discoveries by now.
In physics you still have quantitative predictions done in addition to checking the null hypothesis (which I why I said that was superfluous), if NHST adoption continues these will slowly be deprecated and replaced by only checking the null hypothesis. Seriously go read some old psychology before NHST destroyed it, they were developing quantitative theories of learning and everything [1,2]. Then go look at it now (statistically significant this and that with basically no progress for 60 years...).
Also, the significance level, constraints on p-hacking, etc are adjusted so that discoveries occur at the "right" rate. I'm not sure what triggered he move to 5 sigma in physics but it was probably something like "this seems too easy" so people started carefully double checking and getting conflicting results.
[1] https://www.tandfonline.com/doi/pdf/10.1080/00221309.1934.99...
Nevertheless, I suppose what you would suggest is just setting limits to start with and looking for a bump there instead. That's not a bad idea, and some (Gross, e.g.) have suggested setting limits even in discovery papers.
I don't see how this addresses any of the issues, it just makes it more expensive (ceteris paribus). As I already wrote:
"A more stringent significance level just means it costs more money or the data is much cheaper to collect. ...The way it works is simple, the null model acts as a strawman. The confidence in rejecting the null model is somehow translated into support for the researcher's favorite theory. How does changing the significance level address this problem?"
>"Nevertheless, I suppose what you would suggest is just setting limits to start with and looking for a bump there instead."
Exactly, and the more precise the limits, the more meaningful for the theory if its consistent with the data. At the extreme, theories that allow for any value at all (as I understand the case is for every possible observation under string theory) are not supported by the data no matter how it turns out.
Confidence in rejecting the null isn't "somehow translated into support for the researchers' favorite theory", it's translated into support for the existence of a signal, which makes sense since the null is the background-only hypothesis.
After we establish rudimentary support for a model by finding one or more such signals, we can then embark on the process of doing precision measurement to really test that model (as is being done for the SM).
If its only a statistical fluctuation, the p-value due to sequential sampling follows a markov process (random walk) between zero and one (there is no convergence). So starting from a significant result, the p-value would be just as likely to decrease and become "more significant" as increase and "lose significance". The significance cutoff is irrelevant to this behavior, although the random walk will spend less time in these extreme regions to begin with.
Here is an example of the random walk in R. It was my first attempt when going up to 100k samples, after n = 45 the p-value was ~5e-4 (~3.3 sigma), later at n = 36441 it went even more extreme in the other direction, corresponding to a p-value of ~1.9e-6 (~4.65 sigma). Here is the plot and code: https://image.ibb.co/hOHO08/p_markov_chain.png
set.seed(1234)
n = 1e5
a = rnorm(n)
b = rnorm(n)
p = sapply(10:length(a), function(i) t.test(a[1:i], b[1:i])$p.value)
plot(p, type = "l", panel.first = grid())
> min(p)
[1] 0.0005394751
> 1 -max(p)
[1] 1.908121e-06
>"_if_ what you're seeing is a statistical fluctuating"
You aren't you are seeing that, so everything that follows is irrelevant. The background model is imperfect (eg nobody believes in a standard model with no higgs boson, everyone knows it is wrong somehow). In this (realistic) case convergence to statistical significance is guaranteed with enough data. However, in the rare (non-existent?) case where the background model is thought to be perfect, it is a different story.
>"Confidence in rejecting the null isn't "somehow translated into support for the researchers' favorite theory", it's translated into support for the existence of a signal, which makes sense since the null is the background-only hypothesis."
This amounts to claiming that in most peoples minds (including the authors of the paper) the Higgs boson, gravitational waves, etc haven't been "detected with 5 sigma confidence"... How much would you want to bet on this? The null model could be rejected due to a loose cable (ftl neutrinos), but the p-value doesn't distinguish between these explanations. Perhaps there was "a loose cable" in the LHC experiments but nobody looked very hard since the deviation agreed with preconceived notions? The p-value doesn't help you here.
>"After we establish rudimentary support for a model by finding one or more such signals, we can then embark on the process of doing precision measurement to really test that model (as is being done for the SM)."
Why not just skip the first step? Instead collect data to check whatever models that exist. When none of them can explain it, modify or come up with new models until it fits. Then derive new predictions from those and collect new data to check them, repeat.
EDIT: I ran it again and got this: https://image.ibb.co/g9rGL8/p_markov_chain2.png
> min(p)
[1] 0.005265033
> 1-max(p)
[1] 1.453814e-05If you collect, say, 1000 samples of a normally distributed variable and find that the mean (for instance) is 2 sigma from the expected mean, then if you collect another 1000 samples and take the mean of all 2000 samples, you will probably find that the significance decreases, assuming the first result was a statistical fluctuation. This is just [regression to the mean](https://en.wikipedia.org/wiki/Regression_toward_the_mean).
What we don't do is dishonestly drip-feed data until we hit the significance we want, then stop. Data is added in fairly large chunks (generally an year's run at a time), and when a new dataset is added the analysis is developed with (at least the new) data blinded. The dataset is only unblinded when the analysis has been finalized.
You have found a deviation from your null model. That is it.
>"so it's right to claim a discovery."
No, if you knew the null model was flawed to begin with nothing has been learned. Also I wouldn't refer to something like "equipment was malfunctioning" as a discovery. Drawing any conclusions about the actual research hypothesis must be done outside the NHST framework, or else there is a fallacy at play (probably strawman).
>"What we don't do is dishonestly drip-feed data until we hit the significance we want, then stop. Data is added in fairly large chunks (generally an year's run at a time), and when a new dataset is added the analysis is developed with (at least the new) data blinded. The dataset is only unblinded when the analysis has been finalized."
Great, but even if you don't p-hack it still doesn't work unless it really makes sense to assume the null/background model is perfectly true. P-hacking is just more BS on top of an already BS procedure.
In summary: If there is anything wrong with the null/background model at all, the p-value will converge on zero, it is just a matter of collecting enough data. This will happen regardless of whether the theory of interest (research hypothesis) is accurate or not.
We don't know that the null is flawed to begin with. Not in a way that matters to the analysis anyway, otherwise you've picked the wrong null. No one runs analyses to discover the pion for the millionth time.
For BSM work, the way it generally works is this (ignoring limit setting):
- Pick a signature predicted by one or more BSM models (or even none), that either isn't predicted by the SM or is exceedingly rare in the SM. For example, you might predict a new particle that decays in some predetermined way.
- Define a signal region in some combination of variables where you expect to find a signal corresponding to this signature. This region is blinded, and you don't look at it until after fixing your entire procedure.
- Estimate the background in this signal region with a combination of Monte Carlo simulations and extrapolation from outside this region (data-driven backgrounds). This obviously has both statistical and systematic uncertainties, and a lot of the hard work is in getting this right. For instance, you might use independent Monte Carlo generators, or define control regions to check your estimation in.
- "Open the box" and look at the signal region data. If you see an excess over the estimated background, calculate the significance. Even here, if you were looking for a new particle, for instance, and the excess isn't a localized bump, this would be an indication that the background estimation may be flawed.
- If the significance is over 5 sigma, you have a discovery.
As you can see, you aren't using a null hypothesis that you already know is flawed. Your null is specifically "no signal exists". A positive deviation that isn't a statistical fluctuation is by definition a discovery of a signal.
Things like malfunctioning equipment go into the uncertainties if they occur in a way that the researchers have considered (which often means uncertainties are set conservatively if the equipment behaviour is poorly understood). If they occur in a way that no one considered, there's no statistical trickery that's every going to compensate for that. If we just went straight to setting limits, we would still see a deviation from what's expected there if equipment malfunctioned in a manner that faked a signal.
> In summary: If there is anything wrong with the null/background model at all, the p-value will converge on zero, it is just a matter of collecting enough data.
In summary: Assuming the background estimation is correct, the p-value will only converge to zero if a true signal exists. If the background estimation is wrong, then the background estimation is wrong and this is a problem with the background estimation, not hypothesis testing.
I'm by no means claiming the system is perfect, but it isn't systematically flawed the way you seem to be claiming.
For one more point, we pick the null hypothesis so that we can when we collect data we can calculate probabilities of that data assuming the null hypothesis is true. So, what null hypotheses we can consider are both enabled and constrained by the assumptions to calculate the probabilities given the data.
That is an awful reason to choose a null hypothesis. You should choose a null hypothesis you derived from your theory so you can see if your theory is capable of making accurate predictions.
The role of the test in a scientific theory was deeper than my statement!
In the main, novel hypothesis test I invented and published, there was more than just what I outlined and you criticized, more along the lines you mentioned: That "more" was something like "if the next observation is a long way from anything we've seen before, then maybe it is an anomaly?". The actual work of the math in the paper showed, intuitively, what "a long way" meant and, in particular, for the math, how actually to calculate, along the lines of my statement, the hypothesis test false alarm rate. So, in short, in nearest neighbors anomaly detection, with meager assumptions, we actually can calculate false alarm rate and, thus, get an hypothesis test.
And, then, we are not limited strictly to the single nearest but can do some weighted sum over several of the relatively near. Right, such things appear to be necessarily close to multi-variate probability density estimation. But such estimation commonly requires so much data that it is not a good, direct way to calculate or estimate probabilities; curiously we can do well with the calculations without actually getting a local density estimate although in a sense we are close to doing that.
The other remarks here are even better, e.g., get to how to do science, maybe at the LHC or some such. Good and beyond what I tried to say.
Your alternative hypothesis is derived from your theory.
>"In the physical sciences, the usual result of an improvement in experimental design, instrumentation, or numerical mass of data, is to increase the difficulty of the “observational hurdle” which the physical theory of interest must successfully surmount; whereas, in psychology and some of the allied behavior sciences, the usual effect of such improvement in experimental precision is to provide an easier hurdle for the theory to surmount."
https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte...
> I'd suggest starting with Meehl 1967
Your very quote states that this whole idea works for physics, and is describing that it doesn't work in psychology and social sciences. I'm confused by this:
To take the IQ example, if you take the set of all people in a country, for instance, and randomly select two sets of 5 people, you will find that their average IQ differs by some amount. If you instead select two sets of 50000 people, the average IQs should be a lot closer (to each other, and to the true average). This is by definition true, since IQ is by definition normally distributed. If that's not what you see, you have a methodological problem.
>"Because physical theories typically predict numerical values, an improvement in ex-perimental precision reduces the tolerance range and hence increases corroborability."
I don't know the historical details but it would appear that since 1967 the NHST virus has proceeded to begin its infestation of physics. Now physicists are acting more and more like psychologists (focusing on testing a null model rather than the predictions of their models; which is the first phase of the illness) and will reap the same consequences of stagnation, etc. Lowering the threshold to 3e-7 instead of 5e-2 just makes this slower and more expensive.
>"To take the IQ example, if you take the set of all people in a country, for instance, and randomly select two sets of 5 people, you will find that their average IQ differs by some amount. If you instead select two sets of 50000 people, the average IQs should be a lot closer (to each other, and to the true average). This is by definition true, since IQ is by definition normally distributed. If that's not what you see, you have a methodological problem."
I'm not sure why you are talking about standard errors of random samples from a population... He was just giving stats 101 background for philosophers so he could get to this point:
"While no competent psychologist is unaware of this obvious distinction between a substantive psychological theory T and a statistical hypothesis H implied by it, in practice there is a tendency to conflate the substan- tive theory with the statistical hypothesis, thereby illicitly conferring upon T some-what the same degree of support given H by a successful refutation of the null hypothesis. Hence the investigator, upon finding an observed difference which has an extremely small probability of occurring on the null hypothesis, gleefully records the tiny probability number “p < .001,” and there is a tendency to feel that the extreme smallness of this probability of a Type I error is somehow transferable to a small probability of “making a theoretical mistake.”
In addition, there's a difference between saying "p is low so it is unlikely that the null is true" which is absolutely correct, and "1 - p is the probability the alternative hypothesis is correct". Certainly in physics people don't believe the second, I don't know about social sciences.