Comment by graycat | Hacker News Reader

Comment by graycat | Hacker News Reader

Parent Full thread

How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true.

Or, suppose we have some data and want to use it to estimate the value of some number b. It can be that if we can assume something about b, e.g., that b = 0, then we can calculate the probability distribution of our estimate of b. This assumption about b is the null hypothesis. Intuitively it is a hypothesis that there is no effect, that what we thought might have happened didn't, was a null effect.

We can make two mistakes. We can reject the null hypothesis when it is true -- this is called Type I error. Or we can accept the null hypothesis when it is false -- this is called Type II error.

With our null hypothesis, we get and look at the distribution of our estimate of b and see where our actual estimate is in that distribution. The p-value is the probability, from the distribution of our estimate, of getting an estimate as far or farther from our null hypothesis value for b as we did. So, if the p-value is really small, say, 1%, then we can reject the null hypothesis, that is, say that it is false, and be wrong only 1% of the time.

E.g., if our null hypothesis is that b = 0 and our estimate of b is 10 and from the distribution of our estimate a value of our estimate being as far as 10 from b = 0 is 1%, then we reject that b = 0 and conclude that it b is not zero and are wrong only 1% of the time. If the probability of our estimate being greater than or equal to 10 is 1% and we reject the null hypothesis, then we conclude that b > 0.

This is all just hypothesis testing in statistics 101. Will also want to know about the power of a test, the t-test, the F ratio, the chi-squared test, and resampling and distribution-free tests.

There is more detail in

https://en.wikipedia.org/wiki/Type_I_and_type_II_errors

>"How about: The p-value is, in a statistical hypothesis test, the probability of Type I error, that is the probability of rejecting the null hypothesis when it is true."

No, see here: "Of critical importance, as Goodman (1993) has pointed out, is the extensive failure to recognize the incompatibility of Fisher’s evidential p value with the Type I error rate, α, of Neyman–Pearson statistical orthodoxy."

P Values are not Error Probabilities http://www.uv.es/sestio/TechRep/tr14-03.pdf

Okay, I looked at that PDF. I read through several pages and mostly saw lots of discussion of statements the authors claimed were common and wrong. I can agree with the wrong but am less convinced about the common.

I never saw a clean definition of p-value or alpha -- probability of Type I error.

In my study of statistics, p-value and alpha are essentially the same. Maybe a difference: A p-value is what a researcher has in mind before the hypothesis test as a 'cutoff value', say, 5%, for alpha while alpha is whatever the statistical hypothesis test says, for the data and the test, is the probability of Type I error, e.g., 3.141592653 or some such random number. So, since the 3.14 is less than the p-value of 5%, the experimenter rejects the null hypothsis. MAYBE this is the difference, if any, between p-value and alpha, in which case we're talking a triviality.

The paper you referenced also starts to get wound up over beta = 1 - alpha as the power of a test. Okay, but that little equation is the definition of power. So,if have some data and a p-value or alpha in mind, and have several candidate statistical hypothesis tests, say, some parametric and the others distribution-free, etc., then will want to use the test with the smallest probability of Type II error, that is, the largest beta. Okay. Now are done with beta, and no more strain or struggle needed.

A biggie point is the real role of the null hypothesis -- it lets us calculate some probabilities that, otherwise, we would not have assumptions enough to do.

If there was a fight between Fisher and Neyman-Pearson nearly 100 years ago, then I'm sorry, but now I have no sympathy for whatever the heck, if anything, they were arguing about then.

> Maybe a difference: A p-value is what a researcher has in mind before the hypothesis test as a 'cutoff value', say, 5%, for alpha while alpha is whatever the statistical hypothesis test says, for the data and the test, is the probability of Type I error, e.g., 3.141592653 or some such random number. So, since the 3.14 is less than the p-value of 5%, the experimenter rejects the null hypothsis. MAYBE this is the difference, if any, between p-value and alpha, in which case we're talking a triviality.

This is completely wrong: you're confusing the alpha and the p-value. Maybe you meant that alpha (5% in your example) is the preset type I error rate (significance level) and the p-value (here 3.14%) is calculated from the data. But conceptually the alpha and the p-value are different things and shouldn't be mixed. 3.14% is definitely not "for the data and the test, [...] the probability of Type I error".

If you're doing hypothesis testing, you calculate 3.14 which is less than 5 (alpha) and you reject the null hypothesis. If the calculation had yielded 0.69 instead of 3.14 you would also reject the null hypothesis. But you cannot consider the second result to be "stronger" than the first one. They are both below the threshold and that's all that matters.

If you're using p-values as evidential measures, the second result is indeed "stronger" because 0.69<3.14. But in this framework there are no error rates involved.

> The paper you referenced also starts to get wound up over beta = 1 - alpha as the power of a test.

I've not looked at the paper carefully, but it's unlikely. Power = 1 - beta, it is not directly related to alpha. (Maybe it's just a typing error, I'm not sure what is your point. I mention it mostly for the benefit of other readers that might be confused by your definition.)

>"I never saw a clean definition of p-value or alpha -- probability of Type I error."

On re-reading this I agree. They do not make a clean comparison. They do define them though:

p-value: "the probability of the observed and more extreme data given the null"

alpha: "α is the long-run frequency of Type I errors"

To begin with, you appear to have these reversed. Second, the p-value is dependent on the data, it will be different from experiment to experiment. The exact same experiment can give you p=0.001 one time and p=0.32 the next time. Which is the probability of type I error?

Each of 0.32 and 0.001 is the conditional probability of type I error conditioned on the data actually used in finding it. It is common to play fast and loose when are conditioning on something and when not.

This conditional probability is conditioned on the observations we used; since those are random variables, so is the conditional probability. Then the 0.32 and 0.001 are values of this random variable on the two trials. The expectation of this random variable will be the probability of Type I error alpha.

I strained and tried to find a difference between p value and alpha, but in current usage I see no difference. My guess is that people prefer p-value because it abbreviates probability while alpha does not.

Maybe some people want to say that

alpha = E[E[p value|given data]]

but this is not very operational. Maybe it is what people mean in which case, right, alpha is the expected value of p-value.

So, for a given experiment and hypothesis test, can draw a graph with alpha on the X-axis and beta (probability of Type II error) on the Y-axis. Then the graph shows beta as a function of alpha. So, typically the graph runs from (0,1) to (1,0) and is convex. A more powerful test is lower at each value of alpha. A perfect test is just the point at the origin. A trivial test is the one get by ignoring the real data and using a random number generator, and here the curve is just the straight line from (0,1) to (1,0) and is useful for some cases of interpolating when the available values of alpha are discrete, e.g., in resampling plans for distribution-free statistics.

Right, beta and power = 1 - beta don't have much to do with alpha or p-value except for the point that for a given statistical test in a specific context, e.g., distribution of the data and number of data points, there is that curve I described that relate alpha and beta, and, thus, also 1 - beta, exactly. As I outlined, a different statistical test, maybe parametric instead of distribution-free, can have a different curve and more power for a given alpha.

> Each of 0.32 and 0.001 is the conditional probability of type I error conditioned on the data actually used in finding it.

The probability of type I error of what exactly? Could you be more explicit?

For example: we want to perform a test at the 5% significance level. The procedure is simple: we reject the null hypothesis when p<alpha=0.05. If the null hypothesis is true, in 1000 trials on average:

- we get 0<p<0.01 10 times, we reject the null hypothesis

- we get 0.01<p<0.02 10 times, we reject the null hypothesis

- we get 0.02<p<0.03 10 times, we reject the null hypothesis

- we get 0.03<p<0.04 10 times, we reject the null hypothesis

- we get 0.04<p<0.05 10 times, we reject the null hypothesis

- we get 0.05<p<0.06 10 times, we cannot reject the null hypothesis

- we get 0.06<p<0.07 10 times, we cannot reject the null hypothesis, etc.

The type I error of the procedure is (by construction) 5%: when the null hypothesis is true, we have a rejection (false positive) on average 50 times in 1000 trials.

The "type I error rate" is a property of the test, not a property of the outcome. Using your example, we perform the test once and:

- we get p=0.001, we reject the null hypothesis. The Type I error rate of the test is 5%

- we get p=0.32, we do not reject the null hypothesis. The Type I error rate of the test is 5%

Now, could you please fill in the dotted lines?

- we get p=0.001, we [ reject the null hypothesis / do something else]. The Type I error rate of .................. is 0.1%

- we get p=0.32, we [ do not reject the null hypothesis / do something else ]. The Type I error rate of .................. is 32%

Thinking about this more... first of all it should have been, where P(q)=="probability of q":

alpha = E[E[P(alpha|data)|given data]]

Now we can simplify. Say x=alpha, y=data, z="given data", the E operator can be an arbitrary function f, and applying function f twice is denoted as f#. We get:

x=f#(P(x|y)|z))

We know P(a|b)=P(a & b)/P(b) and p(a & b)=P(a)P(b), so

x=f#{[P(z)P(x)P(y)]/P(y)}=f#{P(z)P(x)}

Denote the inverse of f# as F, then:

F(x)=P(z)P(x)

Now, z was "given data" which has probability 1. So,

F(x)=P(x)

In other words, there is method to your madness.

Just routine conditional expectation. For random variables X and Y (under very meager assumptions,

E[X|Y]

is the almost sure unique random variable that integrates like X over all events in the sigma algebra generated by Y.

In particular

E[X] = E[E[X|Y]]

Details follow from the Radon-Nikodym theorem in, say, Rudin, Real and Complex Analysis where Rudin give the von Neumann proof. Details are also in the standard texts on graduate probability by Loeve, Neveu, Breiman, Chung, and a few more.

>"alpha is the expected value of p-value"

Well, you were correct about this above. You may be onto something here in terms of "actual use".

>"0.001[the p value] is the conditional probability of type I error conditioned on the data"

From the paper: alpha= probability of type I error

So, there are two equations:

p value= p(alpha|data)

alpha = E[E[p value|given data]]

Substituting we get:

alpha = E[E[(alpha|data)|given data]]

What is the difference between "given data" and "data"? How do you isolate alpha?

alpha is a number. Type I error is an event.

Reply on news.ycombinator.com