Understand Entropy in One Chart

Thanks for this. I was following along just fine until that last sentence, which caused me to think "nope, I guess I don't follow at all".

This is just the classic problem of mixing technical terms (like information) and using dictionary definitions. Personally, I believe they intend to say the same thing as you. What they are trying to say is that the lower the entropy, the more similar to a Dirac function. In their mind, this means you know exactly what the distribution is and hence "informative". But, as you point out, that just means you already know everything which is the exact opposite of information. In the context of Wordle, guessing a word with 0 entropy would be a wasted guess as you would have all the previous words remaining. That is, guessing a word that has already been guessed. How informative!

> My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name. In the second place, and more important, no one really knows what entropy really is, so in a debate you will always have the advantage.' [1]

[1] https://en.wikiquote.org/wiki/Claude_Elwood_Shannon#:~:text=...,'

There are so many types of entropy. Why not understand it in your domain and then backtrack the more general definition?

That function is simply -xlog(x)-(1-x)log(1-x) where the logs are in base 2. In general, entropy is E[-log(P(X))] for any random variable. Microstates and such are a physics/chemistry centric application and the namesake but mentioning states is never done from an information theory perspective.