No, see here: "Of critical importance, as Goodman (1993) has pointed out, is the extensive failure to recognize the incompatibility of Fisher’s evidential p value with the Type I error rate, α, of Neyman–Pearson statistical orthodoxy."
P Values are not Error Probabilities http://www.uv.es/sestio/TechRep/tr14-03.pdf
I never saw a clean definition of p-value or alpha -- probability of Type I error.
In my study of statistics, p-value and alpha are essentially the same. Maybe a difference: A p-value is what a researcher has in mind before the hypothesis test as a 'cutoff value', say, 5%, for alpha while alpha is whatever the statistical hypothesis test says, for the data and the test, is the probability of Type I error, e.g., 3.141592653 or some such random number. So, since the 3.14 is less than the p-value of 5%, the experimenter rejects the null hypothsis. MAYBE this is the difference, if any, between p-value and alpha, in which case we're talking a triviality.
The paper you referenced also starts to get wound up over beta = 1 - alpha as the power of a test. Okay, but that little equation is the definition of power. So,if have some data and a p-value or alpha in mind, and have several candidate statistical hypothesis tests, say, some parametric and the others distribution-free, etc., then will want to use the test with the smallest probability of Type II error, that is, the largest beta. Okay. Now are done with beta, and no more strain or struggle needed.
A biggie point is the real role of the null hypothesis -- it lets us calculate some probabilities that, otherwise, we would not have assumptions enough to do.
If there was a fight between Fisher and Neyman-Pearson nearly 100 years ago, then I'm sorry, but now I have no sympathy for whatever the heck, if anything, they were arguing about then.
This is completely wrong: you're confusing the alpha and the p-value. Maybe you meant that alpha (5% in your example) is the preset type I error rate (significance level) and the p-value (here 3.14%) is calculated from the data. But conceptually the alpha and the p-value are different things and shouldn't be mixed. 3.14% is definitely not "for the data and the test, [...] the probability of Type I error".
If you're doing hypothesis testing, you calculate 3.14 which is less than 5 (alpha) and you reject the null hypothesis. If the calculation had yielded 0.69 instead of 3.14 you would also reject the null hypothesis. But you cannot consider the second result to be "stronger" than the first one. They are both below the threshold and that's all that matters.
If you're using p-values as evidential measures, the second result is indeed "stronger" because 0.69<3.14. But in this framework there are no error rates involved.
> The paper you referenced also starts to get wound up over beta = 1 - alpha as the power of a test.
I've not looked at the paper carefully, but it's unlikely. Power = 1 - beta, it is not directly related to alpha. (Maybe it's just a typing error, I'm not sure what is your point. I mention it mostly for the benefit of other readers that might be confused by your definition.)
On re-reading this I agree. They do not make a clean comparison. They do define them though:
p-value: "the probability of the observed and more extreme data given the null"
alpha: "α is the long-run frequency of Type I errors"
To begin with, you appear to have these reversed. Second, the p-value is dependent on the data, it will be different from experiment to experiment. The exact same experiment can give you p=0.001 one time and p=0.32 the next time. Which is the probability of type I error?
This conditional probability is conditioned on the observations we used; since those are random variables, so is the conditional probability. Then the 0.32 and 0.001 are values of this random variable on the two trials. The expectation of this random variable will be the probability of Type I error alpha.
I strained and tried to find a difference between p value and alpha, but in current usage I see no difference. My guess is that people prefer p-value because it abbreviates probability while alpha does not.
Maybe some people want to say that
alpha = E[E[p value|given data]]
but this is not very operational. Maybe it is what people mean in which case, right, alpha is the expected value of p-value.
So, for a given experiment and hypothesis test, can draw a graph with alpha on the X-axis and beta (probability of Type II error) on the Y-axis. Then the graph shows beta as a function of alpha. So, typically the graph runs from (0,1) to (1,0) and is convex. A more powerful test is lower at each value of alpha. A perfect test is just the point at the origin. A trivial test is the one get by ignoring the real data and using a random number generator, and here the curve is just the straight line from (0,1) to (1,0) and is useful for some cases of interpolating when the available values of alpha are discrete, e.g., in resampling plans for distribution-free statistics.
Right, beta and power = 1 - beta don't have much to do with alpha or p-value except for the point that for a given statistical test in a specific context, e.g., distribution of the data and number of data points, there is that curve I described that relate alpha and beta, and, thus, also 1 - beta, exactly. As I outlined, a different statistical test, maybe parametric instead of distribution-free, can have a different curve and more power for a given alpha.
The probability of type I error of what exactly? Could you be more explicit?
For example: we want to perform a test at the 5% significance level. The procedure is simple: we reject the null hypothesis when p<alpha=0.05. If the null hypothesis is true, in 1000 trials on average:
- we get 0<p<0.01 10 times, we reject the null hypothesis
- we get 0.01<p<0.02 10 times, we reject the null hypothesis
- we get 0.02<p<0.03 10 times, we reject the null hypothesis
- we get 0.03<p<0.04 10 times, we reject the null hypothesis
- we get 0.04<p<0.05 10 times, we reject the null hypothesis
- we get 0.05<p<0.06 10 times, we cannot reject the null hypothesis
- we get 0.06<p<0.07 10 times, we cannot reject the null hypothesis, etc.
The type I error of the procedure is (by construction) 5%: when the null hypothesis is true, we have a rejection (false positive) on average 50 times in 1000 trials.
The "type I error rate" is a property of the test, not a property of the outcome. Using your example, we perform the test once and:
- we get p=0.001, we reject the null hypothesis. The Type I error rate of the test is 5%
- we get p=0.32, we do not reject the null hypothesis. The Type I error rate of the test is 5%
Now, could you please fill in the dotted lines?
- we get p=0.001, we [ reject the null hypothesis / do something else]. The Type I error rate of .................. is 0.1%
- we get p=0.32, we [ do not reject the null hypothesis / do something else ]. The Type I error rate of .................. is 32%
alpha = E[E[P(alpha|data)|given data]]
Now we can simplify. Say x=alpha, y=data, z="given data", the E operator can be an arbitrary function f, and applying function f twice is denoted as f#. We get:
x=f#(P(x|y)|z))
We know P(a|b)=P(a & b)/P(b) and p(a & b)=P(a)P(b), so
x=f#{[P(z)P(x)P(y)]/P(y)}=f#{P(z)P(x)}
Denote the inverse of f# as F, then:
F(x)=P(z)P(x)
Now, z was "given data" which has probability 1. So,
F(x)=P(x)
In other words, there is method to your madness.
E[X|Y]
is the almost sure unique random variable that integrates like X over all events in the sigma algebra generated by Y.
In particular
E[X] = E[E[X|Y]]
Details follow from the Radon-Nikodym theorem in, say, Rudin, Real and Complex Analysis where Rudin give the von Neumann proof. Details are also in the standard texts on graduate probability by Loeve, Neveu, Breiman, Chung, and a few more.
Well, you were correct about this above. You may be onto something here in terms of "actual use".
From the paper: alpha= probability of type I error
So, there are two equations:
p value= p(alpha|data)
alpha = E[E[p value|given data]]
Substituting we get:
alpha = E[E[(alpha|data)|given data]]
What is the difference between "given data" and "data"? How do you isolate alpha?