RAG Is Simpler Than You Think
lighthousenewsletter.com
lighthousenewsletter.com
I want fuzzy search like 95% of time and then I might consider having additional list of things that can be suggested by vector search.
Volume of documents, size of documents, versioning, frequency of update, documents similar or overlapping information, how much or exactly what you need for the LLM to understand, AI friendly documents, who has access and at what level, blue teaming, red teaming, multi-lingual, does the LLM know the domain language of the user and documents.
I probably missed a few things even with that.
I think there needs to be a linguist version of "what every programmer needs to know about (full?) text search"...
I'm not a linguist and I don't study languages, but I know enough to realize if a text search system is not designed for a particular language, it simply won't work. (As an example, to implement English search in a system for a hobby project, I had to import a US/UK spelling wordlist, and implement the Porter Stemming Algorithm. This is just for "one" language, and probably does not cover the other "English" dialects. Imagine doing a different workaround for every language in existence...)
RAG is actually a very language-agnostic way to work around those issues.
(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)
It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.
It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.
We were pretty invested in using Google at the time for their Geocoding API. The questions that needed answering for our use-case were:
- Does this place actually exist?
- Can it be found on Google Maps?
- Do we (the company/researchers) have an entry for this place already?
It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
It is, if people don't even stop to think if they need synonyms, periphrases, or mistaken neighbours.
As the blog post points out, more often than not you don't, particularly if your primary usecase is to search for technical keywords or codenames.
Vectors and LLMs are great, but there's no magic pill here. If your parts data and config management[1] is all crazy, that's an institutional problem. Buying a crapton of tokens isn't fixing it, unless you're using it to help build an actual formal solution based on good fundamentals.
[1] Such as it is.
i dont believe ppl are building rag for this
What do you actually think people do when using LLMs to build AI coding agents?
You underestimate the ability of people to overengineer things.
On the other hand, your post may contain a good idea: L=instruct_LLM("provide a list of synonyms and periphrases of terms T within context C", T, C); then iter(`grep l in L`). One NN query and a `grep` collection. But again, if one wanted to order the results, it is either through a dumb crierion or through another LLM query - but this could make it extremely costly (requiring either a huge context or a quadratic number of ordering queries).
And, the above `grep` based procedure would remain keyword based and not semantic based, which means that the user must know that it will not be based on comprehension but on the possible results that keyword matching can yield.
(BTW: you made me realize - I had to take "time off" for over half a year... I am sure I missed a lot.)
--
Edit: for clarity: for "full text search" we remain on the interpretation of "searching for literal substrings" - and whether plain user provided keywords list or LLM enriched list based on the former, and whether more or less successful, it remains a syntactic search quite distinct from a semantic one. Having an LLM enrich the original keywords list can be a good idea, but the possibility of misses remains when compared to a properly working semantic search.
You can see it in action if you watch the “chain of thought” text when using coding agents.
https://www.anthropic.com/engineering/contextual-retrieval
This is from two years ago, but I think it's still SotA?
RAG is routing and decision making.
I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings.
I agree with you.
Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system.
It maintains itself.
Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way.
I found my niche with RAG selling and I build them myself.
I take pride in them.
So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need.
And there are regulated businesses where certain information is required to be always factual correct - pricing for example.
Vector search becomes a liability for this use case.
So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others.
I love this kind of stuff.
And there is personal information etc.
Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart.
And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG.
It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product.
I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers.
RAG gives me dbCORE vibes so to say.
Good luck and fun with your RAG systems.
The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.
Before a lot of frameworks existed, you'd see DEVs taking user input on a web form, and then just throwing it directly at the MTA. So spammers could submit email@address\nCC: persontospam@address, and the like.
Now LLMs are a different beast, but you have input validation for LLMs, unique to all other validation methods. Yet there's actually no safe way to ever validate user input for a LLM, except for very rigid input validation on single words. Take the email example above. You'd need a regex to only validate an email address (and that isn't simple), but once you expand it to actually allowing sentences?
The LLM is now input validation vulnerable.
And that means no user input can be used in unvalidated commands.
And then just random hallucinations. I'm curious how the gp managed weirdo LLM behaviour, like out of the blue 'drop table' or accidental select into as opposed to just select.
>On this benchmark, a pure LLM generated an accuracy score of zero. Adding RAG, prompt engineering, and agentic AI raised accuracy to the 10+% range.
[1] Any text-to-SQL benchmark should address difficulties of real-world data stores (acm.org) (21 comments):
https://news.ycombinator.com/item?id=49013995
[2] If You Think You Can Do Real-World Text-to-SQL:
https://cacm.acm.org/blogcacm/if-you-think-you-can-do-real-w...
[3] BEAVER: An Enterprise Benchmark for Text-to-SQL:
Double hard if you're dealing with private market finance customers with their own ideas on what "common" terms mean. I tried to replace the embedding/ingestion pipeline multiple times and nothing I tried was better over a large amount of documents. Performance sucked, the agent was re-writing and re-trying queries over and over until it found what it wanted, and vector search with a little work up front was worlds better (though it was expensive)
The key thing with RAG is to get the right information in the context with as few queries as possible. That requires good recall (ensuring that if it is there it can be found with a reasonable query) and precision (ensuring the best stuff is on top and minimizing false positives).
With search, and by extension RAG, the principle of shit in, shit out applies. Most of what search teams did before AI and RAG is still the best way to optimize the experience with RAG. And if you mess that up, search is not going to be working that well and no amount of AI can compensate for that or only at great cost in tokens and time. So, having an ETL pipeline to pre-process what you index, testing & benchmarking search quality, etc. are all helpful.
The good news is that you don't need that much skills with agentic coding to build something half decent for this. This code almost writes itself. And even a little bit of effort on extracting structure before indexing can make a big difference.
Similar to SEO on marketing pages, we started rewriting product docs around the idea that it will be consumed by a RAG. Mostly by putting a lot of focus on well structured headlines, thinking more carefully about technical terminology vs common human-language questions, occasionally using variations of keywords in the text, etc. This applies to pure LLM consumption too, not just hybrid search.
Once you start tracking what users are asking you learn to adapt the documentation around it. And LLMs can also suggest improvements by comparing questions vs search results vs LLM responses.
If you flatten all that into plain text and then create embeddings, you are effectively throwing out the baby with the bathwater. But on the other hand if you put some effort into normalizing and extracting some structured meta data, you gain a flexibility to do more sophisticated querying that get you more precise results.
You can of course try to fix things at the source, which is a valid thing but usually not that practical when you have a lot of data to worry about.
No - rag is doing search before you call the llm to give it context from some corpus like your helpdesk articles.
Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.
We've all learnt that it's not really communication, and so can be dispensed with.
"using GPT-4o-mini for query rewriting" -> model from 2024, when RAG was trendy, and all the langchain, llama-index, etc, docs mentioned this specific model
and for the record, my last employer was still using gpt-4o and gpt-4o mini last year. and they are an F500 (not that it means anything, just for context).
This is absolutely not a small draft review or edit.
That is where I stopped
So: https://en.wikipedia.org/wiki/Retrieval-augmented_generation
That is so not Web 5.0. Best I can offer is a support widget that pops up and keeps trying to talk to you until you interract with it.
It’s not that I can’t or don’t know how, it’s rather that the expectation should be that a website should… link you to the information it believes to be relevant background. It’s why it’s called a “web”, linking is a core concept.
in this case there was a menu in the next empty table and you saw it but in place of getting it you want the waitress to get it for you. Which is a normal behavior but you could save your time by just getting the menu yourself.
As OP said, simply providing a link to a Wikipedia article, or a glossary, helps widen the audience beyond "IFYKYK."
The NWS knows this and automatically links to their glossary for both acronyms as well as jargon in their discussions. <-- See what I did there? What does NWS mean in this context? If only I had provided a link that would help you know. I very easily could have. I just didn't.
it also serves as a minimum barrier to entry for the masses, which isn't always a bad thing.
if you're reading this stuff, and you can't figure out what kind of RAG that the search engine mentioned is being talked about through context clues, or you aren't clever enough to feed context into the search like 'hackers , computers, rag' as a query -- there is a very high probability that the person will have absolutely nothing constructive to add to the conversation that is about the topics they haven't even yet conceptualized or are aware of.
in that case that slight barrier to entry for the conversation will serve as a tool to produce less work for the moderators and derail less threads into uselessness.
(much like this stupid divergence.)
No. Lowering the barrier of entry to those who are trying, in earnest, to learn about a new topic, to broaden their base of knowledge is NEVER a bad thing. None of us were born with the knowledge to read this (or similar) article. Trying to kick the ladder down after you have climbed it is terrible behavior and absolutely must be discouraged and stopped at all costs.
> much like this stupid divergence
The only thing stupid in this conversation is the insistence that folks who might know less than the author of the linked article or the poster do not deserve to have access to the information.
Given that the stated purpose of the article is to literally explain how simple something is, not explaining that thing seems a bit misleading, no?
I’m gonna get rich when I make a website explaining all the technical concepts in AI. Every article will just say “lol google it”, it’s gonna be great.
The acronym constraint was valid in a pre-LLM world but now you are perhaps 3 clicks in a modern browser from learning.
If I read an article that spelled out Random Access Memory I would definitely treat that as a lay article.
"on the fly embedding" and "Sparse + dense reranking" don't really make sense how they're presented and smell like they came from a long claude-driven conversation after multiple cycles of these hybrid compromises across many turns.
"Recipe 4: On-The-Fly Embedding (The Fresh Data Play)
The insight If your data changes frequently, why pay to re-embed everything?"
This reads like every Claude generated presentation I've seen.
Then, throw it all in BigQuery. Handles all the vector stuff natively.
Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.
I assume other vendors than Google have a similar batteries-included approach you can just plug in.
This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”
If someone gave me a report, in my hands, that said “see ‘it’” I’d also be confused.
Natural language processing could expanded references, but it starts to get tricky. Do you use Graph RAG, embed another version of the chunk that is distinct from the full text version, etc.. Another layer of processing and data to keep in sync if the source dan be updated.
512 tokens ≈ 350–400 words ≈ a long paragraph or two. 1024 tokens ≈ 700–800 words ≈ about a page and a half to two pages.
I would be very surprised if that amount of text is not enough to encode a general topic into the embedding (otherwise, what would be the whole point of them?).
So if there is a meaningful reference in C referring to A (assuming A-B-C consecutive 1-2 paragraphs), I would expect that the content of the two at least superficially resemble each other, and a vector query for one would return both. (And also, if I am including A in the context after retrieval, then I better give some context before-after as well -- and references tend to be local).
But feel free to prove me wrong, I'm mostly just guessing, not even an educated (in the given topic) guess here.
Because you could have clever ideas about vectors related to more paragraphs related in the document structure - but that would multiply the vectors. The index can become much bigger than the corpus.
https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain )
The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.
Yes, embeddings are computationally heavy, but they are not at all complicated and they provide a lot of benefit.
90% of "document" based RAG projects should view semantic search with embeddings as their primary method.
It's very powerful and so easy to implement that you could try it out and discover whether performance would be an issue rather than trying to anticipate it.
> we don’t especially want to say that books on forestry and similar to books on puppies
Embeddings and semantic search add non determinism on top of non determinism. This seems fundamentally cursed. Lexical is much easier to control, iterate and debug. The tools are incredibly mature. Your users will probably prefer it as well.
If you are building a RAG pipeline for your company and are struggling like me, I would recommend this author that has whole series on entreprise documents (start with the one from May 22nd): https://towardsdatascience.com/author/angela.shi/page/4/
Note: I am not the author, just got her article in my newsletter and found it useful.
I had to look it up.
RAG is simply fetching external data (retrieval) and adding it to LLM context (augmenting) prior to generating a final response.
Any time LLMs do a grep or a web search to answer the query, it’s RAG. Many people use vector db for their own RAG implementation bc of the semantic search benefits.
People don't understand that any sort of retrieval before generation is RAG.
Everyone keeps posting articles about how to implement RAG, but I also wonder why there isn’t some sort of skill to help people create a simple retrieval plan, starting with the retrieval methods and connecting them with evals. This could show whether they actually improve the result and make retrieval simpler for any agent, instead of making people start from zero.
After doing some rigorous A/B testing, we dropped indexing. For coding, I think the reason is that a repo is already searchable. Imports, call sites, file and test names, grep gives you cheap yet reliable version of what indexing would do, and the agent can read around a hit to verify it. Chunked retrieval hands the model something that looks right, and it tends to trust that instead of going to look for the actual source. Another thing that I noticed was the most intelligent models like Opus 5 and Fable ignored chunks anyway most of the time for some reason. Possibly perhaps they are trained around not trusting similarity checks for codebases.
Extremely large codebases with docs feel different. You can’t grep for a concept you can’t name. That’s the case where I’d still use retrieval.
(I work on TheGitAI, for disclosure.)
I find this interesting because practically no one is doing RAG on thier personal data which is something I wouldn’t have expected.
Grep falls apart for severely underspecified queries, which is the difficult part of web search. For any given query in web search there can be several millions of candidate results. You can get good results with FTS as well, but just finding phrase matches is inadequate, you need more ranking signals to find relevant results.
When Claude is looking for a function in your code base, it needs to sift through dozens of matches. This is not hard, and anything beyond grep is likely not worth the effort.
their data use policy from july 2026 explicitly mentions embeddings
RAG is one of those things where I can hyper optimize to an absolutely needless degree.
Can we not reward junk like this? Most of the sentences are incomprehensible and provide zero actual argumentation, it's just a list of "whats" with no "whys"
Oh boy...
How is RAG any different from the search systems we've been building before LLMs? Is it the sudden need for everyone to design a search API and engine that's driven this trend?
If so, I'd like to see more design patterns around existing search problems:
- Correcting or backtracking based on feedback.
- Measuring relevance.
- Comparison with task-based pre-written queries. Does every LLM task need a full blown search engine? Why not a tightly scoped domain API for data retrieval?
Let's say it's just "computation packaged as something new". "Trivial things".
What makes it worse, a lot of people in the thread equate vector search with RAG, whereas RAG is the name for anything that model can query so a user doesn't have to copy/paste feed it to the model manually like access to text files is RAG.
Where's the new design tension? Indexes always had to be monitored for freshness and queries have always needed cleaning or parsing.
at least to me that seems the same as https://en.wikipedia.org/wiki/Word2vec for e.g.
"The future is already here—It's just not very evenly distributed..."
If you're looking for feedback, I'd suggest adding a short demo at the end. It would be nice to see you send it a prompt that says, "hey, remember this" and then tell it to recall that memory. Or show what the memories look like on their way to the model. Are the memories added to the context on every turn or only once per conversation?
I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.
I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."
Any info or advice would be appreciated.
step 1: sparse index retrieval (FTS/BM25) - say with k = 10 step 2: re-rank the 10 records using embeddings
the difference in this approach is during step 2, you convert text to embeddings on the fly - when you're running the retrieval pipeline, meaning you don't need to have all of your corpus pre-embedded in a vector db
Still need ~2 years to be replaced.
With agents, the prompting could be dynamic for maximum accuracy for every retrieval.
This absolutely would beat the best of the best embedding-based RAG models.
Nobody uses this now mainly due to speed. An llm retrieval would be 10x or more slower than embedding.
You can try that now
Take some failing cases or bad retrieval from your current system Prompt an llm wisely like a perfect prompt to get what you want and provide it the context to it. And see the results.
For context, you are limited now by models contexts (1m), so mostly you would need to split what you have and prompt twice....or more...and so on
(found of sid.ai so obv biased)
The AI that wrote this might be the master not the writer, as this looks written by AIs.
I will use the author's agents, not read his articles or use him for the job.
It's necessary and would be good for you if you want to learn something systematically.
But for most of the normal issues, we can not rely a lot on it.
RAG is about providing an grounded response, given the actual data in the corpus.
Great article and content, nonetheless!!
I will keep a note of this article for next time I am asked about this topic.
Whole LLM agent tool call with ripgrep gives 99% use cases right lol
Noun, no?