Hacker News Reader

Comment by dist-epoch | Hacker News Reader

Nothing will crash. It might create "surprise".

Example:

Trajectory: >>What is the capital of France? Let me think<<

the KV cache tokens for "let me think" will have baked in them among other things "France", "Paris", "major city", "question"

Now you compact, and reuse the later tokens (suffix)

Compacted trajectory: >>Let me think<<all right, let's see<<

Now the compacted KV cache for "Let me think" will have baked in them "France", "Paris", and the model can be "that is weird, why am I thinking about France and Paris? there is nothing related in the previous context"

View on HN

So, from this follows there must be a point between "pristine cache" and "100% stale cache" where the surprise is small enough the model can recover. Perhaps with extra hint at the end, "your context cache is no fully rebuilt after a change, except confusion".

Bit like a person who just woke up, or was suddenly distracted, and now has some stray thoughts from previous context floating around. We normally dismiss them. Maybe the models can, too, and this could achieve more flexibility with cache management (therefore much lower costs of context management), at the price of slight capability reduction after context management events?

> Perhaps with extra hint at the end

most of this stuff will be "unconscious" the model will have a pull in a particular direction without being aware

> stray thoughts from previous context floating around. We normally dismiss them

It's not that simple, see https://en.wikipedia.org/wiki/Priming_(psychology).

> Priming is a concept in psychology and psycholinguistics to describe how exposure to one stimulus may influence a response to a subsequent stimulus, without conscious guidance or intention

Also, we are AGI, the model isn't, it can't (re)organize its thoughts as easily as we can.

But nobody knows, what will happen is people will experiment with this, you can run the benchmarks, if it works it will be used, if not, well...

However given it's a super-obvious thing to do, drop middle tool calls from cache and keep the rest without re-prefilling, I would guess it degrades performance quite a lot, otherwise the labs would have been doing this already.

Reply on news.ycombinator.com