Hacker News Reader

Comment by TeMPOraL | Hacker News Reader

> The issue is that when you have messages 1, 2, 3, 4, 5. And you want to update messages 1,2,3, because they are in the reserved chunk, then 4 and 5 already contain some baked in information from 1,2,3. This means you cannot modify 1, 2, 3 arbitrarily without changing the architecture

What does "you cannot modify without changing the architecture" mean exactly? I.e. will the runtime crash if you do it wrong, or would it merely risk confusing the LLM?

Because if it's the latter, I wonder if this isn't less of a problem in practice than it sounds.

Humans don't update their memories all at the same time, either. We often end up in inconsistent mental picture that we resolve automatically when it becomes apparent (the brief "pause to think" moments).

View on HN

Nothing will crash. It might create "surprise".

Example:

Trajectory: >>What is the capital of France? Let me think<<

the KV cache tokens for "let me think" will have baked in them among other things "France", "Paris", "major city", "question"

Now you compact, and reuse the later tokens (suffix)

Compacted trajectory: >>Let me think<<all right, let's see<<

Now the compacted KV cache for "Let me think" will have baked in them "France", "Paris", and the model can be "that is weird, why am I thinking about France and Paris? there is nothing related in the previous context"

So, from this follows there must be a point between "pristine cache" and "100% stale cache" where the surprise is small enough the model can recover. Perhaps with extra hint at the end, "your context cache is no fully rebuilt after a change, except confusion".

Bit like a person who just woke up, or was suddenly distracted, and now has some stray thoughts from previous context floating around. We normally dismiss them. Maybe the models can, too, and this could achieve more flexibility with cache management (therefore much lower costs of context management), at the price of slight capability reduction after context management events?

> Perhaps with extra hint at the end

most of this stuff will be "unconscious" the model will have a pull in a particular direction without being aware

> stray thoughts from previous context floating around. We normally dismiss them

It's not that simple, see https://en.wikipedia.org/wiki/Priming_(psychology).

> Priming is a concept in psychology and psycholinguistics to describe how exposure to one stimulus may influence a response to a subsequent stimulus, without conscious guidance or intention

Also, we are AGI, the model isn't, it can't (re)organize its thoughts as easily as we can.

But nobody knows, what will happen is people will experiment with this, you can run the benchmarks, if it works it will be used, if not, well...

However given it's a super-obvious thing to do, drop middle tool calls from cache and keep the rest without re-prefilling, I would guess it degrades performance quite a lot, otherwise the labs would have been doing this already.

Reply on news.ycombinator.com