Hacker News Reader

Comment by Al-Khwarizmi | Hacker News Reader

OK, I think I more or less got it and I can answer my own question... correct me if I'm wrong, but I think the interpretation "What? If out of every 100 random samples, in 95 of them the parameter is in the interval, then surely the probability of the parameter being in the interval is 95% by definition?" is flawed because we are not talking about the same interval every time, right?

If I have a variable that is always positive, then I could have a weird procedure to generate confidence intervals that gives me the interval [-inf,0] 5% of the time and the interval [0,inf] 95% of the time.

This would meet the definition of confidence intervals perfectly, and yet when I get [-inf,0] the real probability of the parameter being in the interval is 0%, and when I get [0,inf], it's 100%.

I wonder how large this discrepancy may be in practice (as this is obviously a made-up extreme case).

View on HN

The key thing to note is that the parameter being estimated is a fixed (but unknown) quantity. Unlike in Bayesian inference, where we assume the parameter (or our belief in it) is random. The distinction is important. We want inference about the parameter, yet because the interval is the random quantity with confidence intervals, we cannot assign probability statements to the parameter.

As a completely fabricated example, suppose the true proportion in a coin flip experiment is 40%. If my confidence interval is [.45, .65], what's the probability that .40 is in [.45, .65]? It's 1. The probability that a fixed, but unknown parameter will lie in any confidence interval will be either 0 (it isn't in the interval) or 1 (it is). The _proportion_ of times the interval contains the true parameter is the level of confidence (95%).

To your always-positive example, that procedure is not particularly weird. There's always a balancing act with CI's about length and confidence level (otherwise, I could choose all reals as my interval and get 100% confidence level). That your [0, inf) interval has 100% coverage means that you could probably shrink that interval so that it has finite upper bound without losing more than 5% confidence. Hard to say without a specific distribution in mind or mild assumptions, but an application of either Markov's or Chebyshev's Inequality would allow you to make really loose bounds with only relatively minor assumptions.

IIRC, in bayesian terms, confidence intervals express `P(X ∈ [A,B] | X=x) = 0.95`, where X is the unknown parameter and [A,B] is the interval, assumed to be some fixed function of the data. Ordinarily I think they expect this to be satisfied for all values x. So this is P = 95% where the parameter is known but the interval is not (because the interval depends on the data, which is not known yet).

On the other hand credible intervals express `P(X ∈ [A, B] | A=a,B=b) = 0.95` (or more generally `P(X ∈ [a,b] | the data) = 0.95`). The latter is what is intuitively meant by "95% probability" of the true parameter being in the interval, because you do know a and b but not the parameter.

The example with random sampling of confidence intervals from {ℝ⁺, ℝ⁻} is indeed a good illustration of the difference.

Reply on news.ycombinator.com