Hacker News Reader

Comment by Dylan16807 | Hacker News Reader

If 95% of the intervals in your set of intervals include μ, and you randomly pick one of them, in what way is that interval not 95% likely to contain μ? Ignoring the frequentist pedantry that "likelyhood is the wrong word", is there a way for a different number to be the correct number?

View on HN

https://link.springer.com/article/10.3758/s13423-015-0947-8#...

https://sami.boo/jaynes/confidence-intervals-vs-bayesian-int... (+ Cauchy example below)

It’s like with test accuracy. Test accuracy is the pre-test probability that the test will give a correct result. But once you have a positive or negative result, which way it turned out plays a part in computing the predictive value. Likewise, once you have computed the interval, the specific bounds you ended up getting can affect the plausibility that they contain the true value.

Your first link is just the pedantry. The second link... I don't see any issue for the question I asked? The confidence interval is supposed to be wrong a certain percent of the time. It being so wrong it disproves itself is funny but the 90% is still 90% isn't it? They just landed in a particularly harsh part of the 10%.

The conclusion they come to is "it is possible to do better in the individual case by taking into account evidence from the sample that the confidence interval method throws away". That means the confidence interval is inefficient, not incorrect.

If you just meant it in the sense that “if I pick one interval at random and don’t look at it, it will (future) have an n% chance of containing the true parameter” then sure. It’s a pre-data statement (“I have an n% chance of sampling data that will happen to generate an interval that contains the parameter”).

But once you have picked one, and you know its bounds (say, [12.1471, 13.8264]), then it’s fallacious to make the post-data reasoning that “because it was picked at random from the set of 90% confidence intervals, it has a 90% chance of containing the true parameter”.

Again, it’s like with medical tests. If a test has 90% sensitivity and 90% specificity, it has 90% accuracy (it will, in 90% of cases, produce a result that matches disease status) – a pre-data statement on the test result (/ on the confidence interval that we will compute). But it does not follow that, if you screen an asymptomatic patient with low prior probability of disease and get a positive result, they have a 90% chance of having the disease – a post-data statement on disease status, given the test result (/ on where the parameter lies, given the interval).

> The conclusion they come to is "it is possible to do better in the individual case by taking into account evidence from the sample that the confidence interval method throws away". That means the confidence interval is inefficient, not incorrect.

It means you know in the individual case that the specific confidence interval does not in fact have a 90% chance of containing the parameter.

I can see the analogy here but a test like that being binary throws things off and that's also super asymmetrical error. Weren't we sampling a gaussian?

It’s the same principle. The “trivial interval” from the first link (the one you called pedantry), which is applicable to a Gaussian and tweakable to have different coverage than 50%, clearly shows that being an n% confidence interval, on its own, doesn’t guarantee the ability to directly translate that n% into post-data inferences. Likewise, in the truncated exponential example, when you compute that confidence interval of [12.1471, 13.8264], you know that the probability that it contains θ is 0%. In the Cauchy example, when you compute [-2.31, 10.31] as your 90% CI from the two samples 3 and 5, you can then compute that it actually has a >99% chance of containing θ (table I).

The trivial example demonstrates that once you pick your confidence interval you don't truly have odds anymore. But if we're judging whether we have the right odds number, what matters is if a different number is correct. And the only coherent way I can think of to interpret "odds" says the odds are 50%. The trivial example survives that challenge.

The examples that actually ruin the number so far have been asymmetrical. Is there a way to do it with a gaussian? Especially if you're trying to make a reasonable internal?

> The trivial example demonstrates that once you pick your confidence interval you don't truly have odds anymore. But if we're judging whether we have the right odds number, what matters is if a different number is correct. And the only coherent way I can think of to interpret "odds" says the odds are 50%. The trivial example survives that challenge.

Sorry, I don’t follow. Let’s say I want to compute a 50% confidence interval for the unknown mean of a Gaussian distribution. I sample two numbers from the distribution, get 9 and 7, compute the interval according to the trivial procedure and get (-∞, ∞). Does the interval (-∞, ∞) have a 50% probability of containing the mean of that Gaussian distribution? I would think it’s closer to 100%.

If what you are saying is “it’s meaningless to talk about the probability of that specific interval containing the unknown-but-fixed parameter” then that’s the purely frequentist view and then you also agree that it’s meaningless to say that [-2.31, 10.31] has a 90% chance of containing the location parameter of the Cauchy distribution that happened to yield the samples 3 and 5. Incidentally, what asymmetry are you referring to in the Cauchy example?

I misread the Cauchy example since I was going too fast, nevermind the symmetry part.

> then that’s the purely frequentist view and then you also agree that it’s meaningless

I'm saying that when you hit "meaningless" you can back up a step to where you actually had randomness and look at that distribution, which gets rid of a lot of these issues.

But after looking at these examples I think it only makes sense in limited circumstances to do that. Like in the trivial example: your final distribution isn't based on the probability of the mean being any particular number. The only probability was back a step and that was 50%.

At this point I still don't think it's objectively wrong to say a particular interval above is 95% likely, but there's too many ways to interpret the statement so nobody should say it is.

The way we're calculating that these intervals are "wrong" is by looking at all the possible parameters that could have given us the samples we got, and checking how often the range contains the parameter. That's a useful calculation but is it the one people expect? I think that depends on the situation. Treating the parameter as being the thing we sample over is misleading, but treating it as fixed is also misleading.

just read a book on Bayesian stats bro. Probably just the intro of most of them will get you to your answer.

I already have my answer.

Reply on news.ycombinator.com