Hacker News Reader

Noob question from me: what’s the difference between accuracy and calibration? A well calibrated market would be more accurate and vice versa, not?

Edit: just found the answer myself: “accuracy measures the percentage of correct predictions out of total predictions, while calibration assesses whether a prediction market's assigned probabilities align with the actual observed frequency of those outcomes”

View on HN

Suppose there are 1000 events and 500 will have outcome A and 500 will have outcome B. If you predict a 50% chance of A for every event you'll be perfectly calibrated. On the other hand, if you predict a 90% chance of a certain outcome and you're right for 800 events, you're not perfectly calibrated but you have a lower Brier score (lower is better).

A forecaster can be calibrated but almost only assign probabilities in the 40--60 % range. This is not as ueful as one assiging calibrated probabilities in the full range.

We try to measure the increased usefulness of the latter with proper scoring rules.

Reply on news.ycombinator.com