Ask HN: LLMs and Information Theory
In what ways do large language models reshape or expand our understanding of classical information theory, particularly around concepts like meaning, entropy, and communication?
You know, it really takes about tens years of thinking about something to really begin to understand it at a fundamental level (which should tell you something about the true pace of human understanding.)
Shannon said his part over fifty years ago now, and we’re still trying to interpret that.
Let’s see, what might LLMs confirm, deny, or help us ponder?
- the correlation between the volume of training materials and the size and accuracy of their output
- the proficiency and deviation of various training techniques
There is so much more of course, it’s questions like yours that get things rolling.