Comment by cyanydeez | Hacker News Reader

Comment by cyanydeez | Hacker News Reader

Parent Full thread

running local models, now with Qwen3.8-Flash-Next, they have 256k, but when they get up there their speed is just too slow. So i've taken https://github.com/Tarquinen/opencode-dynamic-context-prunin... and started improving it. It already worked well to get a lot of mileage out of just taking tool calls, code modifications, etc, and dumping them in favor of a summary.

But they'd still inevitably get to long in the tooth, and context poisoning meant they'd just eventually not be able to stay in the preferred context size, which for me is 64k-128k. So, I extended it with an eviction command and required a ratio. So instead of a summary of work, it now just places a waypoint. The waypoint basically means the context has a semi-coherent context but without all the baggage.

I'm on like day 3 of a single session with 3m tokens removed and still in the sweet spot. So it evicts to beneath the lower limit, compresses to the upper limit, then evicts again.

It's amazing how resilient it is if you give it a good plan. The work flow has basically been:

1. Write up an implementation document for some new set of features.

2. Rewrite the implementation as a TDD document

3. Set it to work.

The only thing I haven't figured out is it likes to stop when it hits the finish line of the subparts, but likely we're going to end up with the master of puppets monitoring these things and just set them to evaluating what they've done.

> taking tool calls, code modifications, etc, and dumping them in favor of a summary

Doesn't that destroy the cache? I find that caching significantly sped up my Qwen, especially on said larger contexts.

Yes-ish. The compressed summary sits atop the cache stack, so if we got to 96k, it'll take say 30k, compress to 10k, and that 66k+10k is the new stack, so 66k is still cached and retrievable.

So it is designed like a heap, where we're taking raw context off the heap, compressing it, and putting it back on the heap. So cache during compression is mostly unperturbed, since we're rarely digging all the way to the bottom of the stack, but that could happen.

Eviction though is cache busting; but again, I'm valuing the session's roadmap as the valuable product and context size slows computation size, so I have to bust the cache to sacrifice immediate re-processing for longer term compute speed up.

Because that's faster than getting to the end of the context (remember, every 1k adds to the compute time of the next 1k). So speed at 200k is much lower than at 100k. It's also local, so I'm only paying time+watts for the trade off. As far as I can tell, speed is not being lost since if I let the context grow, the kv cache doesn't help with the compute throughput.

So, yes, but it's "smart"; we're only busting it at the top of the context, so rebuilding it isn't from the bottom up, it's just at the top. Those summaries sink on the heap until you get to the eviction limit, and then, they're evicted, and we rebuild from some intermediate place in the heap.

The benefit of it all is I can have lots of projects, and keep a single session that tends to have the context necessary to avoid having to write AGENTS.md or other context bloats. Set large implementation goals and come back to them as needed, etc. I've had it running like this for awhile and it seems Qwen3.8-Flash-Next has no trouble understanding the rolling window.

Reply on news.ycombinator.com