Hacker News Reader

Machine learning models don't do well with math that has hard bounds at 1 and 0 (probability). So we use the log of the odds ratio instead. This function goes from negative infinity to positive infinity. And is mappable back to 0 and 1. So as you get closer to infinity log odds, you get closer to 100% certainty that your answer is A.

It's just ln(p/(1-p)) under the hood.

https://en.wikipedia.org/wiki/Logit

Yeah -- the choice of temperature corresponds to the base of the logarithm. When you think of probabilities in the log-odds representation, a bayesian update means adding an increment. https://arbital.com/p/bayes_log_odds/

this is not what language models typically output as logits though. perhaps the term now is being misused and it’s causing a lot of confusion

Basically, logits are the raw outputs of the last linear layer of a neural network just before the softmax(for multi-class classification tasks) or the sigmoid(for binary classification tasks) is applied to the outputs.

Logits are the raw predictions of the model that you then need to normalize to turn into probabilities (that sum to 1). You can do that is with the softmax function.