Hacker News Reader

Comment by jemmyw | Hacker News Reader

That sounds reasonable. The article doesn't dig into it. Does this group actually make enough predictions to have this kind of validation?

View on HN

Yes, serious forecasters predict on hundreds of questions & the relevant websites prominently feature "binary calibration" plots.

This is mine on Metaculus (mentioned in the article):

https://imgur.com/a/d3s67xk

There's still a bit of noise at 205 predictions but you can see a pattern emerging!

But if you can pick what you’re predicting on and pick the percentage you assign to an event, I fail to see how that gives you a calibration.

Because you make lots of predictions, and smooth out the numbers. If the events you give 8% probability actually happen 35% of the time, you aren't as calibrated as you are if they happen 10% or 5% of the time. The calibration numbers measure how much you are off by.

This makes no sense. If I have 1000 predictions and vary my estimates from 0-10%, that’s only 100 samples assuming you round to the nearest whole percent. And there’s no correlation between any of those samples. For example, I could say the probability of a lightning storm tomorrow is 1% and the probability of a war with China in the next year is 1% - calibrating amount those 1% events is clearly non sensical. You could try to calibrate among similar events but then you have no way of estimating that a 10% war between US and China vs a 1% of war between Russia and China.

Basically this is an exercise of garbage in and garbage out.

I think youre kind of right, its not really meaningful to know “im right about things that happen 5% of the time!”. But if a lot of things you bet on structurally happen to be low probability and of the same type (eg you bet on one type of catastrophic event like pandemic, crop disease, or war etc) then its useful

Or you’ve just learned how to optimize your score on the game you’re playing but the game isn’t actually about predicting the future.

Proper scoring rules like Brier mean that any attempt to optimize your score by definition mean that you are improving your calibration.

See https://samotsvety.org/track-record/

The article does dig into it. They have Brier scores.

Reply on news.ycombinator.com