Hacker News Reader

Comment by unglaublich | Hacker News Reader

An LLM is effectively a compressed model of its input data.

Inference is then the decompression stage where it generates text from the input prompt and the compressed model.

Now that compressing and decompressing texts is trivial with LLMs, we humans should focus - in business at least - on communicating only the core of what we want to say.

If the argument to get a new keyboard is: "i like it", then this should suffice, for inflated versions of this argument can be trivially generated.

View on HN

What I hate about this is that often a novel and interesting idea truly needs extra space to define and illustrate itself, and by virtue of its novelty LLMs will have substantially more difficulty summarizing it correctly. But it sounds like we are heading to a medium-term where people cynically assume any long email must be LLM-generated fluff, and hence nothing is lost by asking for an LLM summary.

What a horrible technology.

Not to be overly snide, but I can imagine that almost every person who writes long, tedious emails that wax on and on thinks they have something novel and interesting that truly needs extra space. Also, most novel things are composed of pedestrian things, which LLMs have no issue summarizing sufficiently.

Maybe you can provide an example where this case would occur, and maybe some indication how often you think this would occur.

It seems to be a big problem for AI-generated summaries of scientific research, where the top-performing AIs ignores key details about 25% of the time, and often much worse. The problem is exactly what I said: the research has a new idea which is (by necessity) not going to be in the pretraining data, so the AI ignores or misstates it. I don't expect this problem to be solved anytime soon, it is inherent to how 2020s artificial neural networks work.

https://royalsocietypublishing.org/doi/10.1098/rsos.241776

Edit: I was thinking about the “overly snide” and am reminded of Sam Bankman-Fried:

  I would never read a book. I think, if you wrote a book, you fucked up, and it should have been a six paragraph blog post.

The problem is not snideness, it is arrogant cynicism leading to stupidity.

> If the argument to get a new keyboard is: "i like it", then this should suffice

This seems like exactly what LLMs are supposed to be good at, according to you, so why don't they just near-losslessly compress the data first, and then train on that?

Also, if they're so good at this, then why are their answers often long-winded and require so much skimming to get what I want?

I'm skeptical LLMs are accurately described as "near lossless de/compression engines".

If you change the temperature settings, they can get quite creative.

They are their algorithm, run on their inputs, which can be roughly described as a form of compression, but it's unlike the main forms of compression we think of - and it at least appears to have emergent decompression properties we aren't used to.

If you up the lossy-ness on a JPEG, you don't really end up with creative outputs. Maybe you do by coincidence, and maybe you only do with LLMs - but at much higher rates.

Whatever is happening does not seem to be what I think people typically associate with simple de/compression.

Theoretically, you can train an LLM on all of Physics, except a few things, and it could discover the missing pieces through reasoning.

Yeah, maybe a JPEG could, too, but the odds of that seem astronomically lower.

If you don't design your compressor to output data that can be compressed further, it's going to trash compressibility.

And if you find a way to compress text that isn't insanely computationally expensive, and still makes the compressed text compressible by LLMs further - i.e. usable in training/inference? You, basically, would have invented a better tokenizer.

A lot of people in the industry are itching for a better tokenizer, so feel free to try.

The inverse of this is "AI Loopidity" where we burn cycles inflating then deflating information (in emails, say, or in AI code that blows up then gets reduced or summarized). This often also leads to weird comms outcomes, like saving a jpg at 85% a dozen times.

And each cycle introducing error like a game of telephone

And with no governance in place (like an engine governor) you get cascade failure.

An early article about LLMs called it "a blurry jpeg of the web" for exactly those reasons. It's a good read.

Be more trivial

Reply on news.ycombinator.com