Hacker News Reader

Comment by simoncion | Hacker News Reader

> Training extracts patterns in the data and encodes them as tiny perturbations in a gazillion weights.

I could say similar things about JPEG compression. And -as it turns out- the raw output of both LLM "training" and JPEG compression are equally incomprehensible. You either need a computer program or an enormous amount of time, patience, and careful effort to convert it into a form so you can make any sense of it.

View on HN

No, you can't say that about JPEG compression. I know how the output of a JPEG compression is structured so it is not incomprehensible to me. (I have worked with A/V transcoding and written custom media compression schemes, but anybody can look up how JPEG works.) JPEG performs relatively straightforward, deterministic mathematical transforms that are actually reversible if you do not discard details (but discarding details is what makes it compress better.) JPEG does not extract patterns from a billion images of, say, cats and encode that into weights that represent the concept of a "cat". Or dogs, or cars, or faces.

LLMs do, and they encode so many concepts and the relationships between them into so many weights that it is a literally incomprehensible blob of floats. They are not just a storage format, and there is no way to recover the original content verbatim from these weights, except for a very small handful of extremely popular works like Harry Potter.

As another example, JPEG will discard details from the original image, and when decoded, will display missing details as blocks. But it will never hallucinate output that never existed in the input data. Like, a JPEG of a cat can never randomly be decoded into an image of a dog.

Me:

  And -as it turns out- the raw output of both LLM "training" and JPEG compression are equally incomprehensible. You either need a computer program or an enormous amount of time, patience, and careful effort to convert it into a form so you can make any sense of it.

You:

  I know how the output of a JPEG compression is structured so it is not incomprehensible to me.

Looks like you didn't read what I wrote with sufficient care.

> ...there is no way to recover the original content verbatim from these weights...

It's impossible to recover the original content verbatim from a lossy compression system. Systems that don't have this property are called lossless. Also, consider the report from the end of August at [0].

> But it will never hallucinate output that never existed in the input data.

Odd... I'm pretty sure that the blocky compression artifacts I see in this JPEG on my desktop weren't in the scene that I set up to capture in that photo. Maybe I need to get my eyes checked?

[0] <https://infosec.exchange/@zzt@mas.to/117134157775929932>, with original challenge at [1]

[1] <https://mas.to/@zzt/117122289150514171>

No, I didn't misread, I literally don't need any computer program (maybe other than a hex editor) to decipher a JPEG image. I could walk you through each part of the image format and tell you how it impacts the output rendered. An interesting exercise to try is to give a JPEG to an LLM and ask you to walk through each byte ;-)

Nobody can say the same about LLMs, because nobody has managed to decipher them yet. This is an active area of research (Mechanistic Interpretability.)

Your post is conflating lossy discarding of information with extracting abstract concepts from information and encoding them into weights. This is why those weights do absolutely nothing until you run a prompt through them. On the other hand, obviously any media file can be decoded by itself without needing a prompt.

Those blocky compression artifacts you see are not hallucinations, they are the image viewer just filling in for missing details. You don't even need lossy compression for this; take a lossless image like a BMP and zoom in, you'll see those blocks again!

Interestingly, image viewers typically do this via interpolation of adjacent pixels, but a lot of hallucinations are actually the result of extrapolation, the opposite mechanism. Which is why LLMs can create output that was NEVER in any input data anywhere (hence the term "hallucination"!)

And if you think lossy compression is sufficient to avoid the "verbatim" requirement of copyright claims, you're welcome to explore the legality of selling transcoded versions of copyrighted content ;-)

I'm not sure what those links are in relation to?

> I could walk you through each part of the image format and tell you how it impacts the output rendered.

I'd be quite impressed if you could correctly hand-decode a five-meg JPEG in less than a workday. You do get that I'm not talking about having an understanding of the file format, but actually being able to convert the encoded data into human-readable [0] output?

> Your post is conflating lossy discarding of information with extracting abstract concepts from information and encoding them into weights. This is why those weights do absolutely nothing until you run a prompt through them.

I can play that game too. The JPEG process extracts perceptual shorthand from information and encodes that into "quantized coefficients". These "quantized coefficients" do absolutely nothing until you run them through a "reconstituter".

Most things sounds quite high tech when burdened with new jargon. It's something you inevitably learn if you work at a Big Software Company for long enough.

> Those [incorrect claims and assertions] you [receive] are not hallucinations, they are the [LLM] just filling in for missing details.

FTFY

> ...take a lossless image like a BMP and zoom in, you'll see those blocks again!

I take it you've never seen a highly-compressed JPEG?

> ... if you think lossy compression is sufficient to avoid the "verbatim" requirement of copyright claims...

Quite the opposite. It's why I even bother bringing up the fact that LLMs are the output of lossy data compression programs.

> I'm not sure what those links are in relation to?

Go back and re-read the paragraph that referred to them, and then consider it how it and the posts relate to the quote that sits right before it. Someone who suggests that they can quickly and accurately hand-decode a non-toy JPEG file definitely has the capacity to read and understand ten-ish Mastodon posts.

Here's a hint to prime your intuition pump: The linked posts are about plagiarism generated by LLM-based tools.

[0] ...in the case of picture data, "convert into human-readable output" means "turn the data back into a picture"...

Eh, a small JPEG is a 100 or so bytes anybody can decode by hand it in ~15 minutes. A 5MB JPEG would be the same 15 minutes repeated and reveal nothing interesting.

>I can play that game too. The JPEG process extracts perceptual shorthand from information and encodes that into "quantized coefficients". These "quantized coefficients" do absolutely nothing until you run them through a "reconstituter".

But the "quantized coefficients" represent only color information of pixels, where are the abstract concepts of the things depicted in the image? I mean, you can't just dismiss salient details as "high-tech jargon" and replace it with completely unrelated alternatives ¯\_(ツ)_/¯

And I do mean abstract concepts. E.g. if it's a picture of a cat, do any of the quantized coefficients map to the concept of a cat, in that they can identify or regenerate any cat regardless of the infinite variety of pixels that can depict an infinite variety of cats?

You know, like they found artificial neurons to recognize arbitrary cats in arbitrary pictures in this research from 2012, which was a precursor to what LLMs are doing today: https://research.google/pubs/building-high-level-features-us... -- that represents the abstract concept of "cat."

Note that it's not even "high-tech jargon" it's very much a plain english description of what is happening inside LLMs. The rest of your comment would be addressed once you internalize that difference ;-)

Reply on news.ycombinator.com