Hacker News Reader

Comment by kqr | Hacker News Reader

What's funny about this is as we get better at text prediction, we revise our entropy estimations downward. The last I heard was 2016, at around 6 bits/word of EU-produced English. I'm sure it's lower now.

View on HN

This is a good point, it happens because newer models are able to use larger and larger context - long range correlations in text. This amounts to finding more redundancy in the source text, driving down the estimated entropy.

Language models are related to text compression, except that the compression is lossy, not trying to be able to recreate the source text exactly but rather to be able to recreate something that is statistically similar.

How is that not just K-complexity? A larger model is a larger interpreter. The entropy is just moving from the stream to the state machine.

For what it's worth, GPT-2 got PTB perplexity of around 20, which translates to about 4.3 bits/word. The later language models are presumably better but they report bits-per-character or bits-per-token which doesn't convert straightforwardly into bits per word.

For a layman, can you elucidate whether this is about the entropy of its output or input?

Kind of neither, but it's more like the entropy of its output than its input. LLM's can be used to generate text but fundamentally what they do is calculate probabilities of a token given previous tokens. Given a context like "I saw a", the LLM gives a probability to each token in its vocabulary within that context. It's really a probability distribution p(token | context). You can sample from that distribution to generate text, or you can use it to get the probability of some fixed text that you feed into it, which is the probability that the model would have generated that text (at temperature 1).

The PTB perplexity numbers are based on the probabilities assigned to some text from the Wall Street Journal that used to be used as a standard test set in language modeling. It's 2 to the power of the average log probability of each word in the text under the model. So perplexity of 20 corresponds to around 4.3 bits of log probability per word, or another way of thinking of it is that the model will guess the correct following word in 1/20 cases.

The reason I say this isn't exactly the output entropy is that, since perplexity is evaluated on a fixed text, it might not correspond to the kinds of texts that the model would generate purely on its own. In other words when you're looking at some fixed txt, the contexts you get in p(token | context) might not be anything like a context that the LM would generate itself. This was a problem people pointed out a few years ago---back then it looked like LM generations were somewhat higher-entropy than what you measured looking at a fixed text. I don't know if the modern huge LMs have this same problem.

It seems plausible that the entry of written English is decreasing over time. As it becomes cheaper to 'publish' the median effort spent in writing is decreasing.

Will ChatGPT drag this downward even more?

I couldn't resist asking it about the word entropy of its training corpus and quite predictably it wasn't able to understand my question and instead answered similar questions. However, it stuck quite firmly to a figure of 10--12 bits per word! I'm fairly sure that refers to the vocabulary entropy and not word entropy of English.

This is like forcing a human to answer questions about brain anatomy.

You’ll be lucky to get anything reasonable at all.

Yes, the GPT's are literally trained to minimize (cross-)entropy of text. You can see the bits-per-byte of text for lots of models here: https://crfm.stanford.edu/helm/v0.2.0/?group=targeted_evalua...

I was wondering the same thing. Even the prediction as I type this in is pretty good. It seems like LLM's could be even better.

Reply on news.ycombinator.com