Hacker News Reader

Comment by Hermel | Hacker News Reader

> Because what the p-value is really telling you is the probability that on repeating the experiment you would see a result as or more extreme than what you observed.

That's incorrect. If you perform the exact same experiment twice, the chances are exactly 50% that the first result is more extreme than the second one (neglecting equal outcomes).

This illustrates nicely how hard it is to properly explain the p-value. Things would be more intuitive if people would report confidence intervals. I prefer "x is with a likelihood of 95% between a and b" a thousand times over "we measured x=c and reject the null hypothesis with 95% probability".

View on HN

> the chances are exactly 50% that the first result is more extreme than the second one

The chance is 50% if you condition only on the information you have before doing either experiment. But once you've done the first experiment, the chance of a more extreme result given what happened the first time may be much more or much less.

(Extreme example: Your experiment consists of rolling ten ordinary 6-sided dice. The null hypothesis is that they're fair dice, fairly rolled, in which case you expect a total not too different from 35 pips. All the dice come up 6. It is not now true that if you run the experiment again, you're as likely to get a more extreme result as you are to get a less extreme one!)

> confidence intervals [...] "x is with a likelihood of 95% between a and b"

But that isn't what a confidence interval means! A 95% confidence interval [a,b] means "If we ran the experiment lots of times, using the same method of computing the interval [a,b] each time, then in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run".

(What you described is what Bayesians call a "credible interval". Of course that interval depends on your prior.)

> in 95% of runs (in the long run) the true value would be in the interval [a,b] obtained on that run

So 95% of the runs produce an interval that includes the true value. Doesn't that imply that when doing one run, there is a 95% chance of the resulting interval including the true value? I.e. the probability is 95% that the true value is in the interval?

Not really. For example, following a procedure which behaves well in the long run and provides 95% coverage you could get a confidence interval that contains only impossible values. Consider also the following example: you want to estimate x from two random numbers taken uniformly from the interval[x-1,x+1]. The interval defined by the two random numbers is a 50% confidence interval (half of the times x will be in the middle of the two). However, for some specific realizations (when the distance between the numbers is grater than one) you will be able to say with certainty that the interval contains x. See also https://github.com/richarddmorey/ConfidenceIntervalsFallacy

Thanks, you are right. I thought I understood confidence intervals, but apparently I did not.

kgwgk's explained why the answer is no, but let me offer a slightly different perspective.

The situation is similar to the one we discussed earlier where you do the same experiment twice. Before you do the experiment, your estimate of Pr(true value is in confidence interval) is 95%. But after you do the experiment, you have extra information: you know how the experiment turned out. And this may lead you to a different estimate of Pr(true value is in the confidence interval).

"x is with a likelihood of 95% between a and b" -- that is not what a confidence interval means. Confidence intervals are about as confusing as p-values and don't really solve the problem.

Statistics are just plain confusing all around.

No, my description is correct, though I left out "assuming the null hypothesis is true."

The equal probability you are describing is for the p-value itself being more extreme, not the result.

Yes, adding the clause with the null hypothesis corrects your statement.

Reply on news.ycombinator.com