Comment by berkes | Hacker News Reader

Comment by berkes | Hacker News Reader

I've been pondering on something related: can an LLM be a chat?

Some models are reproducible, in that the same prompt will generate the same output. Say that we could wire up such a model to generate some code.

In that case, we could create a prompt that generates, say, an entire codebase, or a large piece of text. The prompt (or really, the tokens) would then be the compressed version of the codebase or the text.

I am not talking about an "AI agent", but really a model that we call in a reproducible manner. Preferably one call, with one prompt. An agent could just run `git clone` to "decompress" a codebase, which conflates the idea of compression. If that were compression, then the "compressed version of the git kernel" would be a single line of text: `git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...`. I am really talking about having an LLM re-generate text based on a prompt.

Does that make sense? I can imagine that this is highly impractical and inefficient. But would this count as "compression" at all?

You're describing https://bellard.org/ts_zip/ ("Text Compression using Large Language Models") ?

Man,did that page load fast. It made me realize how slow the rest the (my) web is.

You sound a bit confused.

A large language model itself (the network) give you the probabilities for the next token given some prefix of tokens so far. You can use arithmetic coding to go from these probabilities to a deterministic compression / decompression algorithm.

When you use an LLM to generate text, you sample from that probability distribution. You can use a true random sample. Or you can make it trivially deterministic by using a seeded pseudo-random-number-generator or you just pick the highest probability each time. But that's all a red herring; really, what you want is arithmetic coding.

https://en.wikipedia.org/wiki/Arithmetic_coding

>I've been pondering on something related: can an LLM be a chat?

A chat?

>I am not talking about an "AI agent", but really a model that we call in a reproducible manner.

An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.

>compressed version of the git kernel

The git kernel, got it.

>But would this count as "compression" at all?

Yes. The decompressor is several tens of gigabytes though.

>An LLM is just as deterministic as any other computer program. For identical inputs (which includes the PRNG seed) it produces identical outputs.

This is not really true in practice because of multi-threading and out-of-order execution. Mathematically equivalent orderings of operations are not equivalent when dealing with floating point values, so most practical LLM implementations end up being non-deterministic.

"An LLM is just as deterministic as any other computer program" is not refuted by pointing out hardware limitations that would affect any other computer program implemented at similar scale (weather forecasts, or even just computing the average of a large stream of sensor readings).

I'm not really trying to 'refute' the original statement. Certainly, an LLM is just doing some calculations that can be done deterministically in principle. However, I think it's worth pointing out that there are practical barriers to doing those particular calculations both deterministically and efficiently. People who worry about LLM output not being reproducible aren't necessarily misunderstanding what an LLM is doing; they are responding to a real feature of most practical LLM implementations.

Agreed.

If you wanted to and had enough engineering effort to spare, you could run an LLM deterministically at relatively small impacts to performance.

One approach is to make sure you run things in the same order. Another is to change your operations so that more of them become associative or even commutative.

See eg the paper 'A Lattice-Based Approach to Deterministic Parallelism' for some interesting ideas on the latter.

This was tried many times in the past for images, even before LLMs.

https://imalogic.com/blog/2024/06/03/image-compression-decom...

Sounds like an interesting way for future OS included apps to be distributed.

Like when you click the Calculator button on your android, it wouldn't actually exist yet, your click actually prompts it into existence. But naively that has problems because you don't want a different UI every time. There's something to your idea.

Someone did it!

https://youtu.be/7NfyZhV1dKM?is=YOUXCHuFUiPdlD0p

It makes a lot of sense. I thought about it in the context of pull requests or change sets: if the text-to-code process is reliable, why don't you give me prompts instead of code? Code becomes just an intermediate representation.

Reply on news.ycombinator.com