Comment by Al-Khwarizmi | Hacker News Reader

Comment by Al-Khwarizmi | Hacker News Reader

Parent Full thread

I understand p-values, but I always have real problems understanding the thing of 95% confidence interval not meaning 95% probability of the true parameter being in the interval. I once grasped it, but then I forgot the reason. And now I look at this paragraph:

"the problem is that, for example, a 95% confidence interval does not indicate that the parameter of interest has a 95% probability of being within the interval. Rather, it means merely that if an infinite number of samples were taken and confidence intervals computed, 95% of the confidence intervals would capture the population parameter"

And I say, "What? If out of every 100 random samples, in 95 of them the parameter is in the interval, then surely the probability of the parameter being in the interval is 95% by definition?"

Where was the catch? I remember there was one (which is enough for practical purposes because it means I won't say in a paper that the probability of the parameter being in the interval is 5%) but I feel dumb for not being able to see at first glance something that is supposed to be basic statistics...

OK, I think I more or less got it and I can answer my own question... correct me if I'm wrong, but I think the interpretation "What? If out of every 100 random samples, in 95 of them the parameter is in the interval, then surely the probability of the parameter being in the interval is 95% by definition?" is flawed because we are not talking about the same interval every time, right?

If I have a variable that is always positive, then I could have a weird procedure to generate confidence intervals that gives me the interval [-inf,0] 5% of the time and the interval [0,inf] 95% of the time.

This would meet the definition of confidence intervals perfectly, and yet when I get [-inf,0] the real probability of the parameter being in the interval is 0%, and when I get [0,inf], it's 100%.

I wonder how large this discrepancy may be in practice (as this is obviously a made-up extreme case).

The key thing to note is that the parameter being estimated is a fixed (but unknown) quantity. Unlike in Bayesian inference, where we assume the parameter (or our belief in it) is random. The distinction is important. We want inference about the parameter, yet because the interval is the random quantity with confidence intervals, we cannot assign probability statements to the parameter.

As a completely fabricated example, suppose the true proportion in a coin flip experiment is 40%. If my confidence interval is [.45, .65], what's the probability that .40 is in [.45, .65]? It's 1. The probability that a fixed, but unknown parameter will lie in any confidence interval will be either 0 (it isn't in the interval) or 1 (it is). The _proportion_ of times the interval contains the true parameter is the level of confidence (95%).

To your always-positive example, that procedure is not particularly weird. There's always a balancing act with CI's about length and confidence level (otherwise, I could choose all reals as my interval and get 100% confidence level). That your [0, inf) interval has 100% coverage means that you could probably shrink that interval so that it has finite upper bound without losing more than 5% confidence. Hard to say without a specific distribution in mind or mild assumptions, but an application of either Markov's or Chebyshev's Inequality would allow you to make really loose bounds with only relatively minor assumptions.

IIRC, in bayesian terms, confidence intervals express `P(X ∈ [A,B] | X=x) = 0.95`, where X is the unknown parameter and [A,B] is the interval, assumed to be some fixed function of the data. Ordinarily I think they expect this to be satisfied for all values x. So this is P = 95% where the parameter is known but the interval is not (because the interval depends on the data, which is not known yet).

On the other hand credible intervals express `P(X ∈ [A, B] | A=a,B=b) = 0.95` (or more generally `P(X ∈ [a,b] | the data) = 0.95`). The latter is what is intuitively meant by "95% probability" of the true parameter being in the interval, because you do know a and b but not the parameter.

The example with random sampling of confidence intervals from {ℝ⁺, ℝ⁻} is indeed a good illustration of the difference.

"The probability of this parameter being in the interval" is what's called a credible interval, and it will coincide with the confidence interval if you assume a uniform prior, that is, if before running the experiment you figure every result is equally likely.

In general, frequentists just don't like to talk about the properties of this particular sample, only about long-term frequencies – hence the name. Why? Because they object to the idea of probability as a degree of belief rather than as an objective measure, and given that attitude the statement that "there's a 95% probability the parameter is in this interval" doesn't make any sense: either it's in the interval or it isn't.

> it will coincide with the confidence interval if you assume a uniform prior, that is, if before running the experiment you figure every result is equally likely

Huh? Unless you can bound the set of potential results, this isn't possible. Say I want to estimate the half-life of some material (bounded below, but not above). A uniform prior doesn't exist. How will the credible interval relate to the confidence interval?

Reply on news.ycombinator.com