Hacker News Reader

Comment by whatshisface | Hacker News Reader

Although my failure to see an elephant on the table does not rule out completely that there could be an elephant there, it does limit the possible size of the elephant to a few micrometers. Failure to reject the null hypothesis does in fact provide evidence against the other possibilities, so long as "other possibilities" are understood to mean "other possibilities with big effects."

I don't see why a scientist at a conference who's saying that two groups are the same has to be heard as claiming, "we have measured every electron in their bodies and found that they have the same mass, forget about six sigma, we did it to infinity." Instead they could simply be understood to be saying that the two groups must be similar enough to not have ruled out the null hypothesis in their study.

View on HN

That's the thing. P values don't prove that anything must be. They simply say that if rerunning the experiment again, it would be surprising to get a different result. Conversely, if you don't find "statistical significance" it definitely doesn't mean there isn't a difference. In practice, it might (often) mean the study didn't have enough samples to find a relatively small effect, but the layperson making decisions (do I allow right turn on red or is that dangerous?) may not get that nuance. A book that really helped clarify my thinking on this is _Statistics Done Wrong_ by Alex Reinhart.

Edit: remove "interpret" from last sentence to clarify

> They simply say that if rerunning the experiment again, it would be surprising to get a different result.

That interpretation subtly suggests that the P-value gives an estimate of the likelihood of the alternative hypothesis being true.

> They simply say that if rerunning the experiment again, it would be surprising to get a different result.

P values don't tell you the chances of getting the same result, only with the chances of getting the same result by chance.

That's not what OP is saying though. Let's say p=.001 - That means we're confident the results are not due to chance.

If we repeat the experiment and get a different result, then we need to be looking into confounding variables and testing methodology. Just because the P-value is low doesn't mean there's no fundamental flaw in the experiment used to find the p-value.

P values don't talk about the result, they only talk about the probably of the null hypothesis being correct.

You're only allowed to say either nothing happened or it's unlikely nothing happened, but it doesn't say what the "unlikely nothing happened" means.

> P values don't talk about the result, they only talk about the probably [probability?] of the null hypothesis being correct.

I would say it’s the other way around.

The p-value says something about the result (how likely this result would be if the null hypothesis were true).

It doesn’t say anything about the probability of the null hypothesis being true.

> how likely this result would be if the null hypothesis were true

It's how likely the result is to occur by chance if the null hypothesis is true. A positive result can occur for lots of other reasons even if the null hypothesis is true, and the p-value doesn't tell you anything about how likely you are to get a certain result if the null hypothesis is true (or false).

> It's how likely the result is to occur by chance if the null hypothesis is true. A positive result can occur for lots of other reasons even if the null hypothesis is true, and the p-value doesn't tell you anything about how likely you are to get a certain result if the null hypothesis is true (or false).

I don’t think your comment makes sense.

Given a parametric model and a particular value of the parameter (i.e. the null hypothesis) one can calculate the sampling distribution of the data.

Therefore under the null hypothesis the model gives a well-defined probability distribution for the data and you can tell how likely you are to get a certain result.

There is no room for “other reasons”.

Let's imagine the simplest case where the scientist never actually ran the experiment in the first place and just made up their data.

Looking at the p-value of a study doesn't tell you anything about how likely the study was to have been based on fabricated data.

When we say “how likely you are to get a certain result if the null hypothesis is true”, one should understand “the null hypothesis is true” as “the data is generated by a process perfectly described by the model, including a particular value for the parameter”.

I agree that if the data is made up the results of the study and the statistical analysis based on the results will have no relation whatsoever with the fact that the null hypothesis was or wasn’t true.

The p-value tells you just how likely you are to get a certain (or more extreme) result if the data generating model is indeed correct and the null hypothesis is true.

We agree that the p-value doesn’t tell you anything about how likely it is that the study was based on fabricated data, or how likely it is that the model is correct or how likely it is that the null hypothesis is true.

The p-value doesn’t tell us anything about the real world. It’s a probability conditional on a hypothetical model.

> They simply say that if rerunning the experiment again, it would be surprising to get a different result.

Not really. A low p-value says that it was surprising to get the result that you got, assuming that the null hypothesis is true. And if the null hypothesis is true it would be surprising to get again the same result (i.e. a result as extreme). If the null hypothesis is not true, the result would not be so surprising (or maybe more, if the true effect is in the “wrong” direction).

The result we got gives some evidence for the null hypothesis being false, but if the null hypothesis was very very likely to be true before it may still be very likely to be true afterwards. In that case it wouldn’t be surprising to get a different result if the experiment is performed again.

Illustration: I roll a die three times. I get three ones. P<0.01 (for the null hypothesis of a fair die and the two-tailed test on the average). This is not simply saying that if I roll the die three times again it would be surprising to get something other than ones.

I roll a die three times. I get three ones. P<0.01 (for the null hypothesis of a fair die and the two-tailed test on the average).

Hmm. At a glance, that doesn't seem right. Yes, the chances of rolling 3 1's is 1/(6^3), but if we only rolled once and got a single 1, we wouldn't have any reason to suspect that the die was unfair. So maybe we should only consider the second two repetitions, and conclude with p ~ .03 that the die is unfair? Otherwise, consider the case that we rolled a 1, 5, 2 --- certainly we shouldn't use this series of non-repeated outcomes as p < .01 evidence of an unfair die?

If the die is fair, the average score will be 3.5. One can define a test based on that value and reject the null hypothesis when the average score is too low or too high.

The sampling distribution for the average can be calculated and for three rolls the extreme values are 1 (three ones) and 6 (three sixes) which happen with probability 1/216 each. Getting three ones or three sixes is then a p=0.0093 result.

You raise a valid point. This is clearly not the best test for detecting unfair dice, because for a die which has only two equally probable values 3 and 4 we would reject the null hypothesis even less often than for a fair die! (In that case, the power would be below alpha, which is obviously pretty bad.)

Yes, this immediately reminded me of this book. To add an example (from the book): after implementing turn-on-red in a few places, a study was conducted to check if it increased the rate of accidents. The study found no statistically significant effect, so turn-on-red was rolled out statewide. Unfortunately, the study failed to detect an effect not because there was no effect, but because the sample was too small to confirm that the effect was statistically significant. Now that we have more data, it turns out that turn-on-red does increase the rate of accidents.

Read it in full here: https://www.statisticsdonewrong.com/power.html#the-wrong-tur...

It doesn't limit the size of the elephant.

Any observation is consistent with an infinite number of models. Eg., your sight is defective, ie., in many cases: your sample is biased, not big enough, etc.

And that A correlates with B, or fails to, to "some significance" is consistent with any causal relationship between A and B.

> Surveys of hundreds of articles have found that statistically non-significant results are interpreted as indicating ‘no difference’ or ‘no effect’ in around half (see ‘Wrong interpretations’ and Supplementary Information).

>Eg., your sight is defective, ie., in many cases: your sample is biased, not big enough, etc.

You can't expect a statistical test to comb through your experimental procedure to check whether or not you did it right. (Least of all, to comb through your personal medical history. ;) )

>Any observation is consistent with an infinite number of models.

I'm afraid that slips below the "get out of bed in the morning" pragmatism bar. Although it's a provocative idea, concluding that conclusions are impossible because you could be making a mistake or hallucinating would lead to, literally, not getting out of bed in the morning, if you took it seriously.

At the end of the day you just have to hope that the human beings reading your paper will be able to tell whether or not you did the experiment right. Any statistical test just walls off one way (or a few ways) in which you could fool yourself.

>Surveys of hundreds of articles have found that statistically non-significant results are interpreted as indicating ‘no difference’ or ‘no effect’ in around half (see ‘Wrong interpretations’ and Supplementary Information).

What we really want to avoid here is requiring every paper to have a twelve paragraph Sophist disclaimer about how knowledge is impossible. ;) Just because the text doesn't say "oh and by the way there could be gnomes in the LHC that are fooling us to entertain themselves," doesn't mean that the author wouldn't begrudgingly acknowledge it as an epistemic risk.

Well your "pragmatism bar" is higher that all future science.

It isnt a trivial sceptical point.

Does CO2 cause climate change, or climate change cause C02, or are they both caused by a hidden factor (one of many) or do they co-vary accidentally (ie., no joint cause)?

No "statistical significance" of any kind can decide between them, yet this is how it is used. There is no way to do so with statistics; it requires repeated experimentation and refutation of all plausible causal models until the one remains stands as "the most plausible".

"Statistics" is being used in place of the scientific method which it cannot replace. Covariation is not causation: the covarying of any number of variables is consistent with any number of causal models. No "test" decides.

The evidence for anthropogenic climate change involves a lot more than correlation, so maybe that isn't the best example. In any case, even if everyone on Earth is misinterpreting p-values, that's not a problem with p-values.

The point of statistics isn’t to attain absolute certainty or for the statistical test to make a decision.

The point is for people to make a decision using probability as one available decision aid, and then deciding whether to place a bet.

"concluding that conclusions are impossible because you could be making a mistake or hallucinating would lead to, literally, not getting out of bed in the morning, if you took it seriously."

Not if you don't have free will, which is apparently the conclusion that a lot of scientists seem to be coming to recently.

Haha, then allow me to rephrase. A brain whose molecules react that conclusions are impossible will no longer chemically get out of bed in the morning.

Why can't you act in ways that are at odds with your conclusions?

When I look around me I see people doing this every day.

This is why we report statistical power. It does in fact put probabalistic limits on the size of the elephant. In effect, we can make a statement like "at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table". It's not reasonable to interpret the results of a single study without consideration of power.

> It does in fact put probabalistic limits on the size of the elephant.

Emphasis added (obviously). This word was missing from the analysis of whatshisface (GGP). While all of our models are probabilistic, and it would be silly to constantly describe everything this way in normal parlance, a discussion of how we want to go about building and interpreting models of the world is exactly the domain where this needs to be explicitly stated. You're not wrong here, but neither is GP. GGP is wrong, though not provably so (again, obviously).

> In effect, we can make a statement like "at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table".

The "in effect" disclaimer makes this statement arguable, but fundamentally we still can't say that. We can only say whether or not an elephant was detected, since detection (in this sense) is subjective. We can't say whether elephants actually exist or not. Perhaps all tables have invisible elephants on them, or perhaps all elephants are hallucinations shared by multiple humans through crazy coincidence. If 50% of elephants that actually exist on tables are invisible, and 10% of visible elephants are not detected due to error, then only 45% of studies will detect an elephant even though it will still seem like 90% to best of our knowledge. Since we don't know what percentage of elephants are currently detectable we need to be even more ridiculous and say something like "we guesstimate that there is a 99ish% chance that at least 90% of studies like ours will detect an elephant larger than 20 microns if it is actually there on the table".

Edit: s/whathisface/whatshisface

Yes, okay. There are implicit assumptions in scientific studies ( e.g.: there are no invisible elephants, or the study we are doing is actually related to the question we are investigating! ). Power calculations refer to the model and not to whether it is the correct model. To some extent we routinely worry about certain types of "hidden from observation" problems: we have zero-inflated poisson models, or we worry about what happens if there is a limited susceptible subpopulation ( that could deplete over time differentially depending on treatment, etc ). But it is not correct to suggest that if a study of whatever power does not find an effect, then a huge effect and no effect are equally plausible.

I have seen outrageous examples of "accepting the null hypothesis" many times, but many negative result studies have great value and even a single negative study can provide evidence against a very large effect.

* There are implicit assumptions in scientific studies ... Power calculations refer to the model and not to whether it is the correct model.*

True, but this crucial assumption needs to be kept in mind when using the study to inform yourself of the state of the world. Too often, it's forgotten that there are two questions that need answering for the study to be relevant and, only one of those has a number associated with it.

In that sense, Bayesian statistics, where this is explicit, are less misleading because they actually draw attention to the fact that we don't know that the model is correct.

Agreed.

Interesting point about Bayesian methods, whenever there are potential flaws or additional uncertainty, it's better if they are more explicit to prompt thoughtful interpretation.

It's still really convoluted and removed from the actual elephant. From a frequentist perspective, the elephant either is or is not on the table, and if it is then it has a certain size - not random quantities subject to probabilistic statements.

If you're willing to treat probability as a measure of uncertainty, then you'd be right at home as a Bayesian.

This thread is a prime example of the danger of p-values, which measure the likelihood of the data if you assume the null hypothesis. This is very different than the probability that the hypothesis is true.

Statistical significance doesn't communicate expected bounds on possible effect size though. If the two sigma bounds are hundreds of meters, the failure to see an elephant in the table is completely meaningless. On the other hand, if it's a few micrometers, that tells you a lot.

Methods like gauge R&R can be used to help decide if the measurement system being used in an experiment is sensitive enough to detect an elephant of a given size.

https://en.wikipedia.org/wiki/ANOVA_gauge_R%26R

Yes, failure to rule out an elephant means you probably didn't look hard enough (collect enough data).

But successfully ruling out an elephant is uninteresting if you didn't expect an elephant. The problem is that "statistically significant" sounds impressive, but we shouldn't be impressed.

I guess we need a less impressive term for this? Maybe something like "may have avoided statistical blindness."

I've been suggesting "null-improbable". I'd be happier still if people stopped dichotomizing on this concept, though.

>They could simply be understood to be saying that the two groups must be similar enough to not have ruled out the null hypothesis in their study

The groups need not be similar to fail to rule out the null. It can also be that the measurements are too noisy and too few.

Also on the flip side, if you do reject the null, it doesn't mean the the groups are different. It could also be that you have so many measurements that you are picking up tiny biases in your experiment or instruments.

Null hypothesis testing is almost always too weak of a test to be useful.

To add to this, the main problem with null hypothesis testing IMO is that mathematically you are comparing a infinitesimally small hypothesis: effect exactly equals 0.000000000000... to an infinitely large one: effect is somewhere between 0.00000...00001 and infinity.

A tiny effect is very easy to reject (p<0.0000...000001). I can tell you before running the experiment that any two objects close enough for long enough to be in each other's light cone have at least a tiny effect on each other.

The null hypothesis is hereby rejected for most anything relevant to humans. No need to calculate p-values. Just reference this comment.

Sorta, but people have taken that reasoning and (by this analogy) begun looking for elephants with their eyes closed. In that case, the failure to see an elephant doesn't actually tell you much of anything.

Reply on news.ycombinator.com