Pruning RAG context down to what the answer actually needs

I read “RAG Context” as “the retrieved content injected into the context window”

I agree. We're seeing more variants of "RAG" that aren't semantic at all (e.g. coding agents or simple memory systems that feed summary indexes directly into context).

I think, over time, it's going to become a SQL / NoSQL sort of divide. There will be the right kind of RAG for the job and lots of forcing the wrong kind because the developer doesn't understand the nuances.

RAG, as far as I understand, is a term that came about before LLM tool-calling was as prevalent. Your options were to have an LLM hallucinate up a response, or instead do a [document -> chunk -> embedding -> vector db -> query -> context window] pipeline. I haven't heard anyone talk of LLMs + web search or other tool calls as RAG, even though if you pull apart the semantics the term is applicable. In fact I don't hear people talk about RAG much at all. I suppose much of what people were trying to solve with document chunking/embedding pipelines has been solved with bigger models and tool calls. And along with that change in tooling we have left behind the term "RAG", which leaves it attached to the concept of those pipelines.

RAG is a fancy acronym that basically boils down to: let's give ai agents the super power of information retrieval (aka. search) and "augment" the generation with a list of results by adding that to the context.

The narrow interpretation of this is usually some kind of vector search. Which some people naively treat as magic pixie dust that will make search quality amazing without any tuning whatsoever.

This does not actually work all that well beyond really simple use cases. A well tuned traditional search engine can be surprisingly competitive. And I know people that do pretty complicated things with vector search that usually involve training their own models and spending a lot of effort on testing and validating those are any good.

I've been doing stuff with search for a bit over two decades. AI use cases makes information retrieval more relevant than ever. It's a key ingredient to answering questions for complex, proprietary data. And especially when that data is very complex and unstructured, naive approaches tend to have their limitations. In other words, it can pay off to to sit down and do it properly and think about things like data ingestion pipelines, transforming & enriching data, testing search quality, etc. Most of the success of a good search system usually boils down to getting your data right for indexing and optimizing it for how you are going to query your data.

The good news is that with large context windows, precision (best results are at the top) matters a bit less than recall (the search returns what you need when you search for it) these days. You can compensate for imprecise search by just fetching more results. As long as what you needed appears somewhere in the top 500 or so, you'll be fine. The flip side is of course that you end up adding a lot of noise to your context which might throw the LLM off and in general wastes a lot of tokens. That's why precision is still important.

What the article is proposing is post processing imprecise results to filter out the noise with an LLM to compensate for what is basically not a great search implementation. That can work of course (provided your recall doesn't suck). But it's going to add some cost and latency to searches. And usually, agents do multiple searches.

But if your search is so poor, why bother with vector search at all? Especially dense vector search at scale is not cheap. If you are going to fetch lots of results, just use some cheap lexical searches. Sparse vector search might be a good compromise (higher cost to index but similar performance to lexical search).

> Am I wrong to be somewhat peeved by the use of "RAG" in these contexts?

Probably? The ship has sailed on this one, like it has on "AI" or "crypto". RAG has almost exclusively been used to describe some variation on vector-search + bm25 pushed into the prompt for the last few years.

Yes you are wrong. RAG means retrieval-augmented generation. If you’re generating something and that generation is augmented by some retrieval you’re doing RAG. The retrieval doesn’t need to be from a vector db or even based on semantic similarity.

To be honest as someone working in this space for the past two years, the problem with the “RAG” and semantic search community is it’s mostly vendors and solutions people selling simple, general stuff to product teams.

If you really are into search you probably implement something bespoke for your use case and integrate it into a product directly, and engage with models/infra tools directly rather than through the products in the space.

If you understand how “semantic retrieval” and other search tools are implemented in practice they feel almost embarrassingly primitive to give such fancy names, or pay for through tools that just implement really basic post-filtering. The entire space had the rug pulled out from under it once “agentic search” took off and most major LLM vendors started integrating web search and tool calling into their products. There is still a lot more interesting stuff you could do with customized rerankers/embedding models, and search algorithms, or small models specialized for agentic search/retrieval, etc but the userbase is big companies that realistically don’t need anything more than a list of tech support document titles that a cheap LLM can select from. So “RAG” is basically a sales shibboleth for that type of stuff now.

The contribution here is the technique (which looks like a nice alternative to reranking) rather than the article or it being some kind of think piece, so does it really matter?

I was able to skim it and go from "lol they reinvented re-ranking" to "oh, that's something more interesting, I should try this one day", and the writing didn't get in the way.

I am being constantly mocked for using there term 'knob fiddling', but I am a British child of the 70's :-)

Yep seems like the pruner and reranker could be combined. That is what i built.

Replacing the reranker with the pruner completely was not feasable since it would have to run on the 150-200 chunks we retrieve before reranking.

I've had good success using a local model for preprocessing (tag extraction) and tuning the search parameters (hybrid search).

Likert scale just doesn't work in LLM evals. My idea of 3/5 is different from your 3 and definitely different from an non-deterministic system's 3.

Shallow dismissal

"RAG is dead" was declared because of ever expanding context windows.

However, resources consumed expand so much with context that I think there is a huge practical barrier there. In the beginning AI services were basically free so nobody cared but this is rapidly shrinking.

Personally I see more in a combination of RAG and live querying during the thinking process (e.g. by tools).

Also I don't think dumping any context that might be relevant into the model really helps accuracy. In my experience models just get lost when they get an overload of irrelevant stuff in their context and start overlooking the relevant parts even if it does fit the window.