Hacker News Reader

Comment by firebacon | Hacker News Reader

Check out these links [0] [1] or google for "p value distribution" or "p curve"

[0] http://www-ist.massey.ac.nz/dstirlin/CAST/CAST/HtestPValue/t...

[1] https://en.wikipedia.org/wiki/P-value#Distribution

I'm not following those links either. How is p uniformly distributed under H0? If you assume H0 then obtaining a p-value near 0 is going to be damn impossible. Whereas obtaining one similarly close to 0.5 is going to be ridiculously more likely.

Am I severely lacking sleep and going crazy or something? Maybe I should check back in like half a day to see what people have said, I feel like I must be completely confused right now because literally nothing I've read so far makes sense to me.

They're right. If the null hypothesis is true, then the probability of getting any p value is equal. Put it another way, the p value is the probability of getting the observed data (or more extreme) under the null. So, under the null, 10% of the time you will get data with a p value of 10% or less; 20% of the time, you will get data with a p value of 20% or less; and so on. And that's the uniform distribution!

Here's an R example to play with:

  pvals <- replicate(10000, {
    x <- rnorm(100)
    y <- rnorm(100)
    t.test(x, y)$p.value
  })
  plot(density(pvals))

That will plot you a nice uniform line on [0, 1].

(NB: I have no idea why OP talked about p values following a normal distribution. That doesn't make sense to me, and I think the post has been deleted.)

So I don't have (or know) R, but I do have access to Mathematica, and this is most definitely not giving me a uniform distribution (how could it?!):

  << HypothesisTesting`
  With[{n = 10000000, dist = NormalDistribution[]},
   Histogram[Last[NormalPValue[RandomVariate[dist, n] - RandomVariate[dist, n]]], 500]]

Why is your x variable though? If H0 is true shouldn't your x be fixed?

I think if you remove the subtraction though then you do get a uniform distribution -- in which case I see what the claim is, yeah. Wasn't really clear to me earlier but indeed, getting p = 5% means you have a 5% chance of getting observations that extreme, so I guess it is uniformly distributed!

You'd get the same result whether you test the hypothesis y > x (with both random variables) or y > 0. Under the null, the p-value is uniform on [0, 1], whatever that null is. It follows from the definition of a p value.

(Disclaimer: I am not a real statistician....)

> How is p uniformly distributed under H0?

That's the very definition of a p-value! The mapping of data to p-values is chosen to have a uniform distribution of p-values when the data is distributed according to the null hypothesis. That's the property that makes p-values interesting.

Reply on news.ycombinator.com