How Compaction Works in Pi
earendil.com
earendil.com
For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration.
Much of the time, I prefer to preserve the history of my conversation instead of summarizing it. I find summarized conversations lead to more frustrating future chats because the LLM misses intent and or context. (Or, the presence of paragraphs and paragraphs of LLM output makes the next token predictor dumber? Unsure.)
https://github.com/spott/pi-task-compaction
I’m still playing with it, but it essentially has the model define a region, and mark it with begin_task, and end_task. End_task also requires a fairly hefty summery of the entire region.
It then pulls that entire region and replaces it with the summary. If the model wants it can actually look into the output that was pruned from the conversation, but in practice the summaries have been good enough it hasn’t needed to.
I’ve been using it with 5.6 sol, and even really long sessions have like 6% context used at the end.
I feel like this is kind of subagents, but it doesn't have the clear task -> work -> return result -> die model, rather it's more closely aligned to how I myself multitask, maintaining several long-lived threads of execution, and having to intentionally "forget" about the other ones when it's time to lock in and do real work on the most important one.
this seems to be the fate with all the big grand 'god' projects people are doing with AI. super epic omega knowledge bases, super epic omega personal platforms, etc. I don't think enough people admit that whatever they're working on has evolved into something they don't fully understand.
https://pchalasani.github.io/claude-code-tools/tools/aichat/...
Sounds cool and it does make sensible decisions optically but I haven’t been able to prove that it is meaningfully better than normal compaction. Building harnesses that do interesting things is a lot easier than building more effective harnesses, I guess.
There are other pruning or compaction strategies you can configure too.
You can also use "/btw" to have side tangent conversations with the current context but with the benefit that it will not bloat your current context after the tangent is done.
I wish other harnesses were prioritising this sort of flexibility with context management, but they're likely more incentivised on optimising their cache rates
Ah, something I needed, but did not know I needed. Comes up ALL the time. I want to know something that is well off the task, but it's the best time to ask due to the specifically loaded context.
Having a second model also iterate the resulting messages and remove low-value tool calls could also be interesting. Especially failed calls which add no value.
That naturally trims a lot of context while removing mistakes from the context to prevent poisoning (ie, every draft but the final negatively contributes in some way — that’s why we edited them).
One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches to the second KV cache which takes newly produced input tokens while KV in first cache is getting replaced with compacted summary tokens. This is a kind of ping pong, so we trade more space for less time. Still experimenting but it looks it works, and nice bonus it improves GPU utilization. Btw I have my own harness and model serving code, but it can be easily implemented in any other harness and model server.
It's like asking the questions needed to get the answer you need from the LLM, can be either an answer for a plan/todo or a task to code/change something.
Here's what I do for each of my sessions:
1. For asides, off-topic work, or repetitive work that has already been done in the session, branch backwards (with /tree) and summarize.
2. If I've exceeded 30% or the 'price-doubling' multi-tier pricing, prune (my custom extension).
3. If I've already pruned and I'm still close to 30%, 'prune all' (more extensive prune).
Definition:
'/prune': Removes ~50% context on a fresh session (not previously pruned)
- Keeps: User messages, normal assistant prose, commands/status markers, extension receipts, model settings, and a plain-text receipt for each tool call.
- Removes: Thinking, signatures, actual tool calls/results, tool output, images, compaction summaries, and other extensions’ state.
- Keeps: User messages, normal assistant prose and conclusions, commands/status markers, extension receipts, and model settings.
- Removes: Thinking, signatures, all tool calls/results and output, images, compaction summaries, other extensions’ state, and any tool-activity receipts created by /prune.
Using these I can keep a session going for weeks (or longer), even with extensive use and almost all the important context is preserved while dumping the less important context. Neither command requires an LLM summarization so they execute quickly.
The how is important, but I've found all of the decisions, question, answers, and results are the most important and the tool calls themselves secondary. When necessary, the tool calls can be deleted without much being lost.
> Extensions can intercept and customize both compaction and branch summarization
https://pi.dev/docs/latest/compaction
Just make an extension (or ask Pi to write an extension for itself) that intercepts compaction and leaves only what you want, or rewrites it in any other way. Should be just a few lines.
It sounds crazy, but it's way cheaper and faster for multimodal models to read text on images than the equivalent text represented as tokens. Text tokens take a lot more bytes to represent internally than you would think. Give it images with rasterized text, and there is no OCR-like intermediate step that turns images into text tokens. It just goes straight into the multimodal context where it's all just linea algebra underneath. The cost is more lossiness than if you gave it the same tokens as text. Which is why you don't just render text prompts and files to read as images.
But compaction is inherently lossy, a summary 1-10% of the original length is going to lose things, by definition. So you keep the things you really need to get exactly right in text token summary, but put a lot more of what you would have just discarded into a PNG. Keeping the past context as text in an image is way less lossy than keeping a text summary of that context.
They peeked inside the layers and found the same neurons were usually activating in the middle layers when fed either text tokens or images of that same text. It just takes the model more time and kv cache to do it via text tokens.
Deepseek was the first to really jump on this: https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSe...
Because juggler has a nested-thread architecture, I had a great ah-ha! moment when I realised that with sub-threads, compaction becomes almost free: you just take the list of items in the conversation, move them all into a new sub-thread, then allow that sub-thread to summarise itself (this summary is what the parent thread sees). Hey-presto, your parent thread is now compacted, and the sub-thread contains all the old messages, so you can browse them if you need to, or just undo the whole operation if you change you mind. This struck me as super-elegant, and I was chuffed that it works.
However.. What I find myself actually doing now is just using the "new conversation" tool to hand over to a new conversation - I just tell the LLM "create a new conversation, and tell it everything it needs to continue this task". It gets you to the same place, but generally I'm not interested in the history, and having a fresh conversation with a new name usually feels neater.
User:'How is auth implemented?' -> [thinking] [codebase exploration with [thinking] in between, 10 file reads, 3 of which were "wrong"] [thinking] -> agent_response
This little exchange contains a WHAT (how auth actually is implemented) and a HOW (where that info is and how to retrieve it). Maybe this question was part of a larger task. I think that whole exchange could be summarised before it enters context, kind of like what happens with subagents. The main thread would then consist mostly of [summaries]. Eventually the context will fill up anyway and we would summarise those summaries again. Alternatively one could maintain a [master_summary], kind of like an internal state. So new [summaries] get integrated directly and the [master_summary] gets updated.
Say, what happens when chain of summaries grows so long, that it still overflows context window. Is summarization runned over the summaries in the context window?
1. What happens if it overflows during assistant's turn, while it makes tool calls? Is it better to make a compaction in the middle of the chain of tool calls, or maybe before a potentially long chain of tool calls? If latter, how to choose the right point for compaction?
2. There should be enough space in the context window for a summary. What if, theoretically, a single new user message already overflows the context window? Or what happens if a summary is too long?
Maybe those are stupid questions, but I'm making a point that there is a room for going in-depth.
My problem is that when in a loop and it’s calling tools, it won’t check how close it is to the compacting limit until the whole loop returns to you for the next prompt. And so if you have a run that could go for hours, it’s a gamble if you’ll OOM or an interrupting compaction breaks context and stops the loop without continuing.
There’s a few extensions that all try to solve this problem, but I’ve found none actually work :(
That said, from reading the pi subreddit I don't see people referencing these much.
I stopped compacting and started making new chats every time and it has saved me an immense amount of context. I'm using deepseek and their cache hit is cheap, so combining this I don't spend more than 4 bucks a week.
I really don't mind keeping these docs up to date if it means I can leave my desk to go do something more important
I pretty much just start a new session whenever i fill the context.
For an EPYC with a 5090 (no layers on CPU) vs an M3 max 128GB, qwen 3.6 27B at 128k context / 7k generation:
Cold: prefill + decode Hot (KV cached)
5090 40s + 2-3m = 3-4 min 2-3 min
M3 Max 128GB 14m + 8-10m = 22-25 min 8-10 min
LLMs are perfectly capable of summarising the conversation without a new system prompt.
This way you have a prefix cache hit on the session and don't pay any more than for a regular prompt.
The prompt is assembled as tool schemas, then system instructions, then message history, and caching works on prefixes of that. So the cost of an edit is not its size, it is the size of everything behind it. Edit a tool definition and you have invalidated the system prompt and the whole conversation with it. Edit the tail and you pay for the tail.
I measured this by accident in a multi agent ablation where the only variable was whether the supervisor sent a fixed tool array or a per-task subset to the worker. Runs paying cache creation, out of 120 each: fixed 0 cache creation, per-task subset 58 cache creation. Under a prompt load that was $0.0382 per run against $0.0230, and on a clean context it reversed because there was no prefix worth caching.
Which is the argument for what Pi does here. A pointer is a tail edit. Rewriting is a head edit.
Their approach is minimalism. If you need something pi doesn’t have, the recommendation is to ask pi about it and have it build an extension for you, or look for a plugin.
That said, try it. You may find you didn’t need everything in those ecosystems.
Not a shill.. I just really like pi.
See: https://github.com/earendil-works/pi/blob/main/packages/codi...
I get it into 1M+ routinely on local models with operations between 50k-85k
The pruning does evict cache but not from the start, and only up till the last time you pruned it, as it's just adding messages on top.
but yes, it takes some additional tests and docs to keep it from just becoming hollowed out on tasks; I'd say about 10-20% of the time is just horribly loses what it's doing.