Comment by nonbel | Hacker News Reader

Comment by nonbel | Hacker News Reader

Parent Full thread

Responding to this but resetting the nesting: https://news.ycombinator.com/item?id=17255181

>"Assuming the background estimation is correct, the p-value will only converge to zero if a true signal exists."

Aren't you assuming the background model is correct and a true signal exists?

>"Your null is specifically "no signal exists"."

No, it is never this. It is an entire model, one assumption of which is "no signal exists". The p-value doesn't care which assumption is wrong, it is only wishful thinking that leads to people focusing on that one.

>"We don't know that the null is flawed to begin with."

Sorry, I don't believe this. Give a concrete example. Do you believe the standard model is 100% correct? If not, then any background derived from it must be assumed to be flawed to begin with.

>"Things like malfunctioning equipment go into the uncertainties if they occur in a way that the researchers have considered (which often means uncertainties are set conservatively if the equipment behaviour is poorly understood). If they occur in a way that no one considered, there's no statistical trickery that's every going to compensate for that."

Correct, there is no statistical trickery to compensate for this. It is a scientific problem, not statistical.

Did you check out that Meehl paper I linked elsewhere?[1] If you take a prediction of a model and test that, messing up the experiment means you get results that diverge from your prediction. If you reverse the logic of science so that you test the "opposite" of your prediction, messing up the experiment yields results that seem to support your model. This is why the prediction of the theory needs to be set as the "hypothesis to be nullified".

[1] https://meehl.dl.umn.edu/sites/g/files/pua1696/f/074theoryte...

You know what's also a straw man? Not assuming good faith in a discussion and obtusely attacking something that isn't actually being done. You've been corrected two times now that the statistical testing methods you've been attacking are in fact not how the experiments are being done and that the choice of null hypothesis that you've been attacking is not at all the null hypothesis that is being tested against.

I know you're feeling very right and vindictive but damn if this isn't tiring to read. Make sure you know what you're right about first before assuming the other thing is wrong just because it matches a few keywords that you've read are bad. Because assuming good faith means that you allow for the possibility they might not be doing it wrong. You still get to make your point but it saves your face a lot.

I'd suggest, go outside, attack a windmill for a few hours, think about what you're actually trying to say, and if it's not "hey I read this article on lesswrong and it says everybody is doin it wrong", you'll probably find you can make your point much clearer.

You seem to have more than sufficient knowledge about statistics to think for yourself on this matter.

>'You've been corrected two times now that the statistical testing methods you've been attacking are in fact not how the experiments are being done and that the choice of null hypothesis that you've been attacking is not at all the null hypothesis that is being tested against."

Quote this. Where? I am sure by "corrected" you mean "someone claimed something I believe to be wrong, so I continued to point out why." Amazingly, this looks like yet more strawman arguing...

Two points:

1. If you're using a data driven background, you're background model isn't the Standard Model but that there isn't something special about your signal region.

2. If you are using the SM as your background, then yes you do believe it to be 100% true, since what you are looking for is evidence that there exists new physics not described by the SM.

In other words, the theory under test is the SM, and it can pass very easily. If it fails despite this, that's evidence that it's incomplete. If it passes, the limit setting stage tests whatever new physics we were looking for, and gives an upper bound on how sure we are that it does not exist.

> "1. If you're using a data driven background, you're background model isn't the Standard Model but that there isn't something special about your signal region."

Sure, it means you assume that during the signal detection everything was working exactly as it was during background data collection. There are going to be other implementation specific assumptions like what thresholds to use regarding environment noise triggers, accounting for sensor drift, etc. These data-driven models are approximations, I doubt the people actually coming up with them really believe they are 100% true. If you give a real life example I will point out exactly where there are issues.

> "If you are using the SM as your background, then yes you do believe it to be 100% true"

I don't know what this is supposed to mean. Of course you are assuming your background is 100% true. Do particle physicists actually believe this though? Not from my reading:

"So although the Standard Model accurately describes the phenomena within its domain, it is still incomplete." https://home.cern/about/physics/standard-model

Yes, I already said the background estimation can be wrong. We can check it by defining validation regions outside the signal region and making sure it correctly describes the data there (where there is no signal) and assign a conservative systematic uncertainty to cover any differences, but yes it can be wrong. This is a background estimation problem though, not a flaw with hypothesis testing. By the way, the signal region data and background data are collected simultaneously, so there can be no systematic difference in experiment conditions between the two.

As for the SM, we are looking for those places where it is incomplete, and to do this we must assume it is complete and look for strong evidence that we are wrong. To date, it doesn't seem like we see though, as far as collider physics is concerned.

Reply on news.ycombinator.com