Hacker News Reader

I don't think you can. If you give every candidate a 1/n probability in winning, but the actual true probability is less than 1% (I'm looking at you, Vermin Supreme[0]), then your calibration is going to be way off. Good calibration is having your 65% chance of victory be right 65% of the time.

[0]https://en.wikipedia.org/wiki/Vermin_Supreme

View on HN

Right, but say in 2016 you gave a 1/3 chance to Trump, 1/3 chance to Clinton, and 1/3 chance to Vermin Supreme. Exactly one of them will win, so your calibration for 1/3 will be perfect. If you do the same for elections with different numbers candidates, you can have good calibration overall without actually producing any information over the base rate.

I disagree. Suppose you have an election with 3 candidates that runs every year. If you predict candidate A win at 33% confidence, but that candidate gets elected every year, your calibration will be way off. Calibration is about aggregating, so you can't really cheat it. Your 33% prediction has to be right 33% of the time - it's really hard to fake that.

Maybe I'm misunderstanding your position, though.

That's right if you only predict the probability of candidate A, but if you are predicting 33% for all three candidates each year, your 33% prediction will consistently be right 1/3 of the time in aggregate.

Reply on news.ycombinator.com