Hacker News Reader

Article says that it _can_ be calibrated, it just isn't out of the box. So the title seems contradicted by the body.

> If you want calibrated probabilities you’ll still need to recalibrate Jev’s probabilities on your own data. The good news is that is cheap. A few hundred labeled examples from your actual data can be enough to fit a Platt scaling on top of Jev’s scores.

View on HN

I suspect they meant it as "they claim it is already calibrated, but it can't be, because there is no universally correct calibration", and not "it is not possible to calibrate Jev", but I read it as the latter at first too.

Yes, good point. I meant that the Jev can't be calibrated for everyone out of the box. I find it important because this is one of their main claims. They literally say

> Outcomes assigned a probability of 0.2 should occur about 20% of the time.

and this is not true

Ah, I see, you mean "Jev cannot possibly be calibrated for everyone as there is always missing context".

I understood it as "It is impossible to calibrate Jev".

Reply on news.ycombinator.com