How to Fix Your Context | Hacker News Reader

How to Fix Your Context

dbreunig.com

93 pointsitzlambda26 comments

> When prompting DeepSeek-v3, the team found that selecting the the right tools becomes critical when you have more than 30 tools. Above 30, the descriptions of the tools begin to overlap, creating confusion. Beyond 100 tools, the model was virtually guaranteed to fail their test. Using RAG techniques to select less than 30 tools yielded dramatically shorter prompts and resulted in as much as 3x better tool selection accuracy.

> For smaller models, the problems begin long before we hit 30 tools. One paper we touched on last post, “Less is More,” demonstrated that Llama 3.1 8b fails a benchmark when given 46 tools, but succeeds when given only 19 tools. The issue is context confusion, not context window limitaions.

High number of tools is a bit of a "smell" to me and often makes me wonder if the agent doesn't have too much responsibility. A bit like a method with so many parameters, it can do almost anything.

Have folks had success with agents like that? I found the fewer tools the better, e.g. <10 "ballpark".

we have success with 39 but we're introducing more focused agents and a smart router because we see the writing on the wall among other things (benefits)

This isn't going to be much of a problem for long. I'm wrapping up on an agent context manager that gives effectively infinite context while producing ~77% better results than naive vector+BM25 baseline on my benchmark suite.

May I ask the rationale for writing your own? Were you using an existing tool that didn't quite fit your needs?

This is an itch I've been wanting to scratch myself, but the space has so many entrants that it's hard to justify the time investment.

Existing off the shelf IR tools are mid, more recent research is often not productionized, and there are a lot of assumptions that hold for agentic context (at least in the coding realm, which is the one that matters) that you can take advantage of to push performance.

That plus babysitting Claude Code's context is annoying as hell.

Thanks.

>That plus babysitting Claude Code's context is annoying as hell.

It's crazy to me that—last I checked—its context strategy was basically tool use of ls and cat. Despite the breathtaking amount of engineering resources major AI companies have, they're eschewing dense RAG setups for dirt simple tool calls.

To their credit it was good enough to fuel Claude Code's spectacular success, and is fine for most use cases, but it really sucks not having proper RAG when you need it.

On the bright side, now that MCP has taken off I imagine one can just provide their preferred RAG setup as a tool call.

You can, but my tool actually handles the raw chat context. So you can have millions of tokens in context, and actual message that gets produced for the LLM is an optimized distillate, re-ordered to take into account LLM memory patterns. RAG tools are mostly optimized for QA anyhow, which has dubious carryover to coding tasks.

> ... re-ordered to take into account LLM memory patterns.

If I understand you correctly, doesn't this break prefix KV caching?

It is done at immediately before the LLM call, transforming the message history for the API call.

This does reduce the context cache hit rate a bit, but I'm cache aware so I try to avoid repacking the early parts if I can help it. The tradeoff is 100% worth it though.

I’m curious about this project (I’m working on something similar). Anyway to get in contact with you?

you can click my spam protected email links on https://sibylline.dev, those should be working now. Any CTA will get me.

This seems to be an important article.

However it uses various terms that I'm not sure of the definition for.

E.g. the word "Tool".

How can I learn about the definitions of these words? Will reading prior articles in the series help? Please advise.

tool (n)

One who lacks the mental capacity to know he is being used. A fool. A cretin. Characterized by low intelligence and/or self-esteem.

Disclaimer: I only cite definitions from Urban Dictionary, but I remain firmly convinced they are correct definitions in context.

You should make a Chrome plugin that fills in Urban Dictionary definitions of first names while you’re on LinkedIn.

Here's my attempt at explaining tool calling:

https://youtu.be/owDd1CJ17uQ?si=Z2bldI8IssG7rGON&t=1330

It's an _incredibly_ important concept to understand if you have even a passing interest in LLMs. You need to understand it if you want to have any kind of mental model for how LLM-powered agents are even possible.

Thank you, I watched it. The key takeaway I got was that the client (browser, I suppose) does the actual usage of such tools. The user hands over control of these tools to the AI (and the tool-use happens in the background so it might look to the user like the AI is the one doing the actual usage of the tools).

Ordinarily:

   you> what's going on?
   > It's going great --- how can I help you today?

Tool calls:

   you> [json blob of available "tools": "ls", "grep", "cat"]
   you> what's going on?
   > [json blob selecting "ls"]
   (you) presumably run "ls"
   you> [json blob of "ls" output]
   > [json blob selecting "cat foo.c"]
   (you) dump "foo.c"
   you> [json blob of "cat foo.c"]
   > I can see that we're in a C project that does XYZ...

The key thing is just: tools are just a message/conversation abstraction LLMs are trained to adhere to: they know to spit out a standardized "tool call" JSON, and they know to have multi-round conversations with sets of different "tools" (whichever ones are made available to them) to build up context to answer questions with.

That's the whole thing.

This is all true, and we've prototyped a number of these things at my current startup. You need to be pretty considered about implementing them.

For a counter-example, consider Claude Code:

- 1 long context window, with (at most) 1 sub-agent

- same tools available at all times, and to sub-agent (except: spawning a sub-sub-agent)

- Full context stays in conversation, until you hit the context window limit. Compaction is automatic but extremely expensive. Quality absolutely takes a dive until everything is re-established.

- Deterministic lookup of content. Claude reads files with tools, not includes random chunks from RAG cosine similarity.

I could go on. In my experience, if you're going to use these techniques 1) maybe don't and 2) turn up the determinism to 11. Get really specific about _how_ you're going to use, and why, in a specific case.

For example, we're working on code migrations [0]. We have a tool that reads changelogs, migration guides, and OSS source. Those can be verbose, so they blow the context window on even 200k models. But we're not just randomly deleting things out of the "plan my migration" context, we're exposing a tool that deliberately lets the model pull out the breaking changes. This is "Context Summarization," but before using it, we had to figure out that _those_ bits were breaking the context, _then_ summarizing them. All our attempts at generically pre-summarizing content just resulted in poor performance because we were hiding information from the agent.

[0] https://tern.sh

What do you mean re Claude Code, "at most 1 sub-agent"?

It only spawns a single sub-agent (called Task iirc), which can do everything Claude Code can, except call Task().

This is different from a lot of the context-preserving sub-agents, which have fully different toolsets and prompts. It's much more general.

Reply on news.ycombinator.com