Text Compression as a Test for Artificial Intelligence (1999) [pdf] | Hacker News Reader

Text Compression as a Test for Artificial Intelligence (1999) [pdf]

aaai.org

37 pointsprude15 comments

I'd be interested to see lossy text compression. That would also give much more scope for AI methods. It turned out that recently Stable Diffusion gives excellent compression

https://pub.towardsai.net/stable-diffusion-based-image-compr...

Funny thing is, on a per pixel basis, Stable Diffusion and jpeg are just as good. Stable diffusion looks much better, though. The reason is that when Stable diffusion lacks the information it just invents what it expects to see there. If you zoom in on the background entire appartement buildings are invented, moved or disappeared.

So, one wonders what GPT-3 could do for text compression. On a per-character basis, I would not expect any miracles. On a generally, the same, kind-of, basis, I'd would expect something special.

A lossy text compressor can be converted into a lossless compressor quite easily by encoding the remaining differences between the lossy reproduction and the original text. The more accurate your lossy compressor is, the less additional information you need to encode the differences. You can get even better results if your lossy compressor is probabilistic, and can compute approximate probabilities for different text continuations. GPT-3 is AFAIK probabilistic and should be applicable... maybe someone has already tried?

But if you're not careful, the delta between the compressed version and the original could be bigger than the savings from the compression.

In my armchair speculation for the case of severe lossy text compression, the delta/diff could easily get there, making a strong compression algo based on something like GPT-3 not really practical for doing lossless.

Instead of storing the differences, use a spell checker. The better written the text with proper grammar rules, the easier for getting it right with a spell checker. Yes, patented.

I hope that lossy text compression never becomes popular, it should ideally remain a technological curiosity. Think of all the damage that can be caused by incompetence combined with usage of lossy text compression in the wrong places.

Human brains have been performing lossy text compression for thousands of years. I think we’ll be ok.

For classical compression benchmarking, you need to include the size of the decoder; GPT can already recite most lyrics and speeches given a single line prompt.

The state of the art in neural language models was evaluated ˜5 years ago, and it was found that standard LSTM's do very well on text compression, when properly architectured and parametrized.

The main reason for excluding lossy text compression in these tests, is that there is no clear path around requiring a panel of human judges, and that evaluation now is subjective (instead of objective).

Perhaps a different route would be to task an AI to compress Wikipedia into a (graph) knowledge base, and then test these AIs on correctly answering "multiple choice"-questions. But then intelligence becomes a proxy for measuring compression, instead of here, where compression is chosen as a proxy for measuring intelligence.

It has been tried

https://bellard.org/libnc/gpt2tc.html

This article doesn't mention the Hutter Prize of course, since it was started a year later, but it looks like Matt has written about it since then: http://mattmahoney.net/dc/rationale.html

Right now the biggest thing holding Hutter Prize back is the shamefully low compute and memory limits: "Restrictions: Must run in ≲50 hours using a single CPU core and <10GB RAM and <100GB HDD on our test machine." http://prize.hutter1.net/

To even begin approaching intelligence, the compute and memory limits probably need to be 100x or 1000x larger.

Strong disagree.

In a completely unrelated field, I'm seeing pretty great results with knowledge distillation to generate AI language models that fit into 20 mio float16 parameters (40MB) and can process 1GB of raw text in roughly 12 hours.

12 hours on a single CPU core, or on a GPU?

12 hours on a Ryzen 5900X with 12 cores. So not quite fast enough yet, but only 3x too slow.

I wonder how the use of static world-prior information is considered here.

Or said in a different way: What stops me from supplying a hard-coded pre-compressed file to circumvent their RAM and HDD limits?

It's stopped by the fact that technically quantity that's being measured in the specific rules (http://prize.hutter1.net/hrules.htm) is size of the compression program plus the size of the compressed output ("zero-input decompressor" which is code+data that can produce the uncompressed output), so you can use static world-prior information but that counts against you twice as it has to be contained both in the "compressor" and the "decompressor".

For most cases the size of the "compressor" is irrelevantly tiny compared to the data, but if your "compressor" is a hard-coded pre-compressed file which it copies to the "decompressor" then the few percent of gains don't outweigh the fact that you've just doubled the size that's scored. It could be useful to include hardcoded priors iff they are very slow to compute but can be expressed in a small amount of storage.

Reply on news.ycombinator.com