Hacker News Reader

Comment by dadrian | Hacker News Reader

You look across unrelated events that you scored similarly. In aggregate, across all things you predict as having an 8% chance to happen, 8% should actually happen.

View on HN

That might give you some confidence. But if you're looking at your performance on unrelated predictions, it won't necessarily detect problems with a given prediction.

I predict there's an 8% chance of one drive in my four-hard-drive array failing in the next year.

I predict there's an 8% chance the next UK election ends without any party holding a clear majority, leading to a labour-lib dem coalition government.

For the first prediction to be accurate, I just need to read the backblaze hard drive stats and multiply.

For the second prediction to be accurate, though? That depends on a lot of factors that are a lot harder to know. For example, would the lib dems be likely to enter a coalition, given how the last one went for them in 2015?

Sure, you cannot expect to reliably detect problems with a given prediction, since it could always be right or wrong due to chance. But also, what would you do if you had the ability to detect problems with a prediction?

If you just want to use a different prediction method that fixes the problem and makes better predictions, looking at aggregate performance over many unrelated predictions does help. Compare how well different methods do, go with the best.

But you could come up with a probability prediction on whether a random prediction from the predictor is accurate.

That sounds reasonable. The article doesn't dig into it. Does this group actually make enough predictions to have this kind of validation?

Yes, serious forecasters predict on hundreds of questions & the relevant websites prominently feature "binary calibration" plots.

This is mine on Metaculus (mentioned in the article):

https://imgur.com/a/d3s67xk

There's still a bit of noise at 205 predictions but you can see a pattern emerging!

But if you can pick what you’re predicting on and pick the percentage you assign to an event, I fail to see how that gives you a calibration.

Because you make lots of predictions, and smooth out the numbers. If the events you give 8% probability actually happen 35% of the time, you aren't as calibrated as you are if they happen 10% or 5% of the time. The calibration numbers measure how much you are off by.

This makes no sense. If I have 1000 predictions and vary my estimates from 0-10%, that’s only 100 samples assuming you round to the nearest whole percent. And there’s no correlation between any of those samples. For example, I could say the probability of a lightning storm tomorrow is 1% and the probability of a war with China in the next year is 1% - calibrating amount those 1% events is clearly non sensical. You could try to calibrate among similar events but then you have no way of estimating that a 10% war between US and China vs a 1% of war between Russia and China.

Basically this is an exercise of garbage in and garbage out.

I think youre kind of right, its not really meaningful to know “im right about things that happen 5% of the time!”. But if a lot of things you bet on structurally happen to be low probability and of the same type (eg you bet on one type of catastrophic event like pandemic, crop disease, or war etc) then its useful

Or you’ve just learned how to optimize your score on the game you’re playing but the game isn’t actually about predicting the future.

Proper scoring rules like Brier mean that any attempt to optimize your score by definition mean that you are improving your calibration.

The article does dig into it. They have Brier scores.

See https://samotsvety.org/track-record/

Doesn't seem right, if a state capital city base chance for a nuclear strike is 1% and you have 195 capitals or so 2 of them would be hit by now.

Maybe your base rate is way off then?

Nuclear strikes on N different capital cities aren't statistically independent events, and that kind of computation only makes sense for statistically independent events.

You're correct it mostly only works for a spread of predictions that don't have such a large influence on each other.

If one capital gets nuked the odds of another one suffering the same fate skyrocket.

Yeah, it's just about possible to imagine Kyiv or Tehran gets nuked in a "limited engagement" that never escalates further. But if, say, Lisbon gets nuked then the whole world is at war.

That sounds a bit like the gambler's fallacy to me.

Reply on news.ycombinator.com