Hacker News Reader

Comment by sarma17 | Hacker News Reader

qq, can you explain a few noob thoughts. Why is surprisal defined as `log (1/p(X=x)`? I'm lacking the intuition for 1. why did someone think to use 1/p? 2. why do we have a log term here.

Log shows up a lot but I don't think I ever really understood when/how do people decided to use it in their formulas. One KL divergence youtube video said, `let's normalize the p(P)/p(Q) using (log(p(P)/q(P)))^N` and then shows how to derived the KL formula. I'd also appreciate if you know how the N (sample size ig) is being used here.

For a lot of math stuff I endup being confused about how people use operations. Like using 1/p(X=x) is like magic to me because I don't understand what the context is when someone thinks about the problem and then decides to do something. What's their thinking process here, what tool or process do I not know about that makes me confused?

View on HN

It's not answering quite the question you asked, but Shannon - who invented much of this stuff - has some really nice practical arguments in the start of his amazing paper that introduced "information theory" [1]. It's a really readable paper, much less intimidating than you might think, and worth a look.

Hartley was (as far as I know) the first person to recognise the usefulness of log probabilities in the context of measuring information [2]. It's a really amazing paper, for several reasons ... but one thing that really strikes me about it is how it's aged: the first half has an essentially timeless presentation of the essence of information, and the second has a now-quite-irrelevant presentation of how to make a better TV. I guess that was the really interesting problem at the time!

[1] https://people.math.harvard.edu/~ctm/home/text/others/shanno...

[2] http://dotrose.com/etext/90_Miscellaneous/transmission_of_in...

Negative log is the only decreasing function which satisfies f(x * y) = f(x) + f(y). If you have two independent events it makes sense that the total surprisal (information content) should be the sum of the surprisals of the two events.

Another way to see it is as a continuous generalization of the idea that you need n bits to represent 2^n equally likely alternatives.

thanks for the reply but I don't think I understood what you said in the context of my question.

I'm stuck on why we care about surprisal as `log 1/p`.

log 1/p is just -log p

I think "information" is a better name than surprisal.

If your distribution has N equally-likely values, `p(x) = 1/N`, and information/surprisal `I(x) = log(N)`. In base 2, this is how many bits are required to specify exactly WHICH of the N values you're talking about.

If `x` is not a single one of the N states but an event consisting of `n(x)` states, then `I(x) = log(N) - log(n(x))`, suggesting it takes _somewhat less information_ to specify this particular state, and it correctly gives 0 if `n(x) = N`, i.e. there's only one state.

Exactly what this "less information" means is vague, but you might think of it in terms of compression: if you compress some stream of data which is sampled from `X` with probability `p(x)`, you could use use the shortest codes (0, 10, 11, etc) for the most common values with some "stop word" to say when the end of a datum is reached. `I(x)` captures this sense in general, but it might only become literally true in the limit of a very large stream of data with a very large dictionary.

> If you have two independent events it makes sense that the total surprisal (information content) should be the sum of the surprisals of the two events.

I think the parent is also asking why we would expect surprisal to be additive rather than multiplicative like probabilities.

Because if two things happen – one totally expected and one very surprising – then on net you’re still surprised.

That's also the case for probabilities.

I think in part because it's really nice to measure information in bits rather than tiny probabilities. And it aligns well with how information is stored.

Convolution?

> Negative log is the only decreasing function which satisfies f(x * y) = f(x) + f(y)

Times a constant!

> Why is surprisal defined as `log (1/p(X=x)`?

Good question. It's because -log P(X) is the number of bits of information I would need to eliminate any surprise about the value of X.

For example, let's say X is the value of an 8-bit register, and each of the 2^8=256 possible values has a probability of 1/256. I would need to learn all 8 bits to know the value of X: so Surprisal(X=x) = -log P(X=x) = -log 1/256 = 8.

Suppose I learned only that X is odd, which is equivalent to learning that the last bit of the register is equal to 1. The probability that X is 1 is 1/2, and so I would need to be provided with -log P(X is odd) = -log 1/2 = 1 bit of information to learn that X is odd.

Information can be thought of as simply the elimination of uncertainty. So Surprisal (uncertainty) is basically a measure of information, but instead of how much information you have, it's how much information you need to eliminate all uncertainty.

There’s some information here about why this was chosen. Specifically it fulfills certain criteria: https://en.m.wikipedia.org/wiki/Entropy_(information_theory)...

Reply on news.ycombinator.com