Hacker News Reader

Comment by sheeeeesh | Hacker News Reader

Grant Sanderson has an excellent video on the same topic [0]. It's part of a series that is ongoing.

[0] Compression is Intelligence Part 1 - https://youtu.be/l6DKRf-fAAM?si=yyLWq8x4sSRkWd98

Thank you for sharing!

Small off topic thing: I recommend removing the bit in the URL from ?si=... and forward, unless you want Google to track every user who clicks this link to your share.

I wonder if the author of the article knew about the series, or do they both just independently came across this topic to talk about it.

Anybody working in the field will be very familiar with these concepts.

…for years. Because it is so apparent if you actually try to look at the problem and what is being solved by it.

The extraction of features from a corpus, the features significant to certain solution, is always and since day zero - compression. As this is the definition of compression - efficient and potentially lossless feature extraction.

It's basic information theory, which has been around since the end of WWII. It's a common topic today because some of its subtle insights are becoming increasingly relevant in our current era of AI, as we learn to understand these black boxes.

common theory. see https://prize.hutter1.net/

And the Hutter Prize for AI which measures how good AI is by measuring how well it compresses data is over 20 years old now just to really drive the point home.

it was vaguely in my understanding of information & intelligence with compression; it was also brought up in several of the initial trials against AI companies where they discussed how the AI is akin to compression.

So they're both sourcing a bit broader zeitgeist.

In the article, she credits the 2023 DeepMind paper Language Modeling Is Compression as the source. She also links to this post from 2015: https://colah.github.io/posts/2015-09-Visual-Information/

Reply on news.ycombinator.com