P-values are inconsistent

johndcook.com

3 points·ekiru··4 comments

How do you do a statistical test against "mu is between a and b?"

If H0 (null hypothesis) is mu = a (and you know sigma = 1 and its normal) then you can pretty easily calculate the p value from an observation x by taking z = x - u and checking that against the normal distribution.

However, how do you do that is H0 is a < mu < b? Do you assume that mu is uniformly distributed between a and b? If so, then H' does not imply H. E.g. if H is true then pr(mu > 0) is 0.5, but if H' is true then pr(mu > 0) = 0.52/1.34 < 0.5.

Or, am I wrong about how to do the hypothesis test?

You use the cumulative distribution function, which is defined as P(X≤x). More specifically, denoting the cdf of the standard normal distribution by 𝛷, P(X̅-a𝜎<µ≤X̅+b𝜎) = P(a<Z≤b) = P(Z≤b)-P(Z≤a) = 𝛷(b)-𝛷(a) for large samples.

Edit: Err, maybe I got that wrong. Doing the above calculation would favor the broader hypothesis (p=0.047 vs. p=0.043).

Well, if we leave the details aside, you do something where you take a probability distribution between a and b right? If thats the case, I think my original point stands, neither hypothesis really implied the other one because they say different things about the probability that x is in certain ranges.

I'm sure there are other problems with using the p value as a measure of certainty (though I don't know them) but this specific criticism seems silly.

I absolutely agree. I can't see anything wrong with my calculation above, which leads me to conclude the blog post or the article it is based on misunderstands – or at the very least miscommunicates – the problem, if indeed there is one.

The actual problem might be that standard statistical tests are difficult to use, but then isn't that why the theory had to be developed in the first place?

Reply on news.ycombinator.com