Hacker News Reader

Comment by Alifatisk | Hacker News Reader

Does it reset at every turn? From my experience in Codex for example, Luna (Max) fills the 256k token window relatively quick. The only thing lowering the context window again is the compaction.

View on HN

I mean the thinking does not bloat the context window because it gets dropped at the next request.

It doesn't. It's called preserved reasoning and every recent reasoning model does it

Sorry, I've realised I was only partially correct.

Gemini[0] for example passes along a snapshot of the reasoning state but it's not the equivalent to keeping all the reasoning tokens in the context.

[0] https://ai.google.dev/gemini-api/docs/thinking#signatures

Edit: Apparently it does take the same space in the LLM latent space so I was wrong.

Yeah, I remember reading about Anthropic's experiments with dropping out different things, and dropping the thinking is ok for compaction but pretty awful while it still fits in the context window. I could imagine doing it adaptively -- take a snapshot before the thinking, have a lesser model detect if there's a lot of spinning going on and reset to snapshot + result if so, otherwise accumulate. It'd also be interesting to have a lesser model rewrite thinking to be more streamlined (let's pretend the AI went directly down the right path on the first try). But it may be a disaster anyway -- isn't the thinking section something like a 1d best-logit slice of something like a 2d process, where the real processing could easily be happening under the visible surface?

I'm not sure if that's right. I only just recently learned that the "snapshots" (aka cached tokens) necessarily contain the entire context history, not just an image of a "state as of the final token" (well, it is the state as of the final token, but that state contains the whole history). So I'm not confident of my grasp of the structures here.

Reply on news.ycombinator.com