> what did i get wrong here?
You don't know how an LLM works and you are operating on flawed anthropomorphic metaphors.
Ask a frontier LLM what a context window is, it will tell you.
You don't know how an LLM works and you are operating on flawed anthropomorphic metaphors.
Ask a frontier LLM what a context window is, it will tell you.
For example, DeepSeek 3.2, which employs sparse attention [1], is not only faster with long context than normal 3.1, but also seems to be better (perhaps thanks to reducing the noise?).
[1] It uses still quadratic router, but it's small, so it scales well in practice. https://api-docs.deepseek.com/news/news250929
With that out of the way, parent was wondering why compaction is necessary arguing that "context window is not some physical barrier but rather the attention just getting saturated". We're trying to explain that 3+2=2+3 and you people are sitting in the back going "well, actually, not all groups are abelian".