Hacker News Reader

Comment by PaulHoule | Hacker News Reader

This 'calibration' is a missing link for machine learning.

A prediction is much more valuable if you know how good of a prediction it is than if you don't.

I'd caution about putting too much stock in calibration, though. It's worth noting that you can achieve perfect calibration just by giving each of n candidates 1/n probability of winning.

I don't think you can. If you give every candidate a 1/n probability in winning, but the actual true probability is less than 1% (I'm looking at you, Vermin Supreme[0]), then your calibration is going to be way off. Good calibration is having your 65% chance of victory be right 65% of the time.

[0]https://en.wikipedia.org/wiki/Vermin_Supreme

Right, but say in 2016 you gave a 1/3 chance to Trump, 1/3 chance to Clinton, and 1/3 chance to Vermin Supreme. Exactly one of them will win, so your calibration for 1/3 will be perfect. If you do the same for elections with different numbers candidates, you can have good calibration overall without actually producing any information over the base rate.

I disagree. Suppose you have an election with 3 candidates that runs every year. If you predict candidate A win at 33% confidence, but that candidate gets elected every year, your calibration will be way off. Calibration is about aggregating, so you can't really cheat it. Your 33% prediction has to be right 33% of the time - it's really hard to fake that.

Maybe I'm misunderstanding your position, though.

That's right if you only predict the probability of candidate A, but if you are predicting 33% for all three candidates each year, your 33% prediction will consistently be right 1/3 of the time in aggregate.

Reply on news.ycombinator.com