Ask HN: Is RAG the Future of LLMs?
It seems to be in vogue that RAG is one of the best solutions to reduce the problem of hallucinations in LLMs.
What do you think? Are there any other alternatives or solutions on sight?
What do you think? Are there any other alternatives or solutions on sight?
The latest connotation of RAG includes mixing in real-time data from tools or RPC calls. E.g. getting data specific to the user issuing the query (their orders, history etc) and adding that to the context.
So will very large context windows (1M tokens!) "kill RAG"?
- at the simple end of the app complexity spectrum: when you're spinning up a prototype or your "corpus" is not very large, yes-- you can skip the complexity of RAG and just dump everything into the window.
- but there are always more complex use-cases that will want to shape the answer by limiting what they put into the context window.
- cost-- filling up a significant fraction of a 1M window is expensive, both in terms of money and latency. So at scale, you'll want to filter out and RAG relevant info rather than indiscriminately dump everything into the window.
What needs to happen is a way to cheaply suspend and rehydrate the memory state of the forward pass after you've fed it a lot of tokens.
That would be a sort of light-weight/flexible/easily modifiable/versionable/real-time-editable alternative to fine tuning.
It's readily doable with the open weights LLM's, but none of them (yet) have the context length to make it really worthwhile (some of the coding LLM's have long context windows, but it doesn't solve the 'knowledge base' scenario).
From a hosting perspective, if fine tunes are like VM's, such frozen overlays are like docker containers: many versions can live on the same server, sharing the base model and differing in the overlay layer.
(a startup idea? who wants to collaborate on a proof of concept?)
Am I understanding what you're describing about the VMs and containers analogy?
I wouldn't be surprised if GPT-4's rumored mixture of many models does something like this overlay management internally.
But frontier models are just too damn good and convenient so I don't think its possible to fully get away from web LLMs.
One of these will remain true until every person has their own pet model which is fine-tuned, on keyup, on all public data and their own personal data. Still, something heinously parametric (like regional weather on some arbitrary date) I struggle to imagine fitting into a transformer.
Edit: I can imagine every user getting a LoRA.
Among other things because it’s way too expensive and narrowing your scope cuts huge costs and isn’t hard to do at a high level
I still think LLMs are the best AI tech/tools since I started getting paid to be an AI practitioner in 1982, but that is a low bar of achievement given that some forms of Symbolic AI failed to ever scale to solve real problems.
I think both projects were super useful, grateful for them. There is a lot of utility tucked away in both projects.
A year ago,I started using LLM APIs in non-Python languages and realized that sometimes building from scratch is better.
That being said, I do use RAG a lot (though generally code most of the implementations myself).
Since you asked about alternatives...
(a) "World models" where LLMs structure information into code, structured data, etc. and query those models will likely be a thing. AlphaGeometry uses this[1], and people have tried to abstract this in different ways[2].
(b) Depending on how you define RAG, knowledge graphs could be a form of RAG or alternatively an alternative to them. Companies like Elemental Cognition[3] are building distinct alternatives to RAG that use such graphs and give LLMs the ability to run queries on said graphs. Another approach here is to build "fact databases" where, you structure observations about the world into standalone concepts/ideas/observations and reference those[4]. Again, similar to RAG but not quite RAG as we know it today.
[1] https://deepmind.google/discover/blog/alphageometry-an-olymp...
[2] https://arxiv.org/abs/2306.12672
[3] https://ec.ai/
Try it out: https://github.com/truefoundry/cognita
That’s RAG. Doesn’t matter that you didn’t use vectors or knowledge graphs or FTS or what have you.
Then the jump from “this whole document” to “well actually I only need this particular bit” puts you immediately into the territory of needing some sort of semantic map of the document.
I don’t think it makes conceptual sense to think about using LLMs without some sort of domain relevance function.
From the video in this IBM post [0], I understand that it is a way for the LLM to check what its source and latest date of information is. Based on that, it could, in principle, say “I don’t know”, instead of “hallucinating” an answer. A RAG is a way to implement this feature for LLMs.
[0] https://research.ibm.com/blog/retrieval-augmented-generation...
The art of implementing RAG is deciding what text should be pasted into the prompt in order to get the best possible results.
A popular way to implement RAG is using similarity search via vector search indexes against embeddings (which I explained at length here: https://simonwillison.net/2023/Oct/23/embeddings/). The idea is to find the content that is semantically most similar to the user's question (or the likely answer to their question) and include extracts from that in the prompt.
But you don't actually need vector indexes or embeddings at all to implement RAG.
Another approach is to take the user's question, extract some search terms from it (often by asking an LLM to invent some searches relating to the question), run those searches against a regular full-text search engine and then paste results from those searches back into the prompt.
Bing, Perplexity, Google Gemini are all examples of systems that use this trick.
There are many tricks to get better context to send to your LLM, and that’s a large part of making the system give good answers.
i am sure there can be newer ways to do prompt injection in an elegant way, but for the most part the llm is either summarizing the injected prompt or regurgitating it.
if the output is satisfactory, it is still more convenient than writing custom rules for answers for each kind of question you want to address.
I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on.
RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimize loss, it is just a similarity score.
Obligatory note that I could be wrong and it's just my armchair opinion
A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things.
As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.
Which is all to say really, data quality is probably the most distinguishing factor in LLM/RAG systems... to get back to OP's questions
I could very well be wrong! But we wouldn't want LLMs to be performing lots of arithmetic calculations via exploiting hidden parts of themselves that do linear regression or whatever, far better to just give them the calculator and get results faster and cheaper. Similarly, we can give them a search engine (RAG) and let them figure it out more efficiently.
“This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation”
I'd assume "large enough" context is the actual goal here, not "virtually infinite".
Rag is not bad attention, rag is the user not knowing the context to give the LLM
Anything that can do RAG is, by definition, a system that wraps an LLM with additional code that performs the retrieval.
It's the difference between ChatGPT (software that wraps a model and can extra features such as tool usage, Code Interpreter, RAG lookup via Bing etc) and GPT-4 Turbo (a model).
In order to use tools they need to be run as part of a system that grants them access to tools, eg via the reAct pattern. https://til.simonwillison.net/llms/python-react-pattern
Obviously I’m talking about next gen models, like gpt5/6.
As you train a model it should build it's knowledge database at the same time. I guess that's how it work?
When an LLM get trained on next word guessing, can it search the web to improve his results?
You wouldn’t say a steering wheel is built into an engine. People just build cars with engines.
Transformer Memory as a Differentiable Search Index:
Both waste compute because you have to re-encode things as text each time and RAG needs a lot of heuristics + a separate embedding model.
Instead, it makes a lot more sense to pre-compute KV for each document, then compute values for each query. Only surfacing values when the attention score is high enough.
The challenge here is to encode global position information in the surfaced values and to get them to work with generation. I suspect it can't be done out of the box but we it will work with training.
This approach has echoes of both infinite context length and RAG but is an intermediate method that can be parallelized and is more efficient than either one.
Do you have a link? Or maybe you misunderstood what I was taking about
The current workflow is to use the embedding to retrieve documents then dump the text corresponding to the embedding into the LLM context for generation.
Often, the embedding is from a different model from the LLM and it is not compatible with the generation part.
So yea, RAG does not pre-compute the KV for each document.
It's 1000x more efficient to give it a look-aside buffer of info than to try to teach it ab initio.
Why do more work when the data is already there?
So you'd still want to use RAG as a performance optimization, even though today it's being used as more of a "there is no other way to supply enough of your own data to the LLM" must-have.
Longer term it gets more interesting.
Assuming we can solve long (approaching infinite) context, and solve the issues with reasoning over long context that LangChan correctly identified[1] then it becomes a cost and performance (speed) issue.
It is currently very very expensive to run a full scan of all knowledge for every inference call.
And there are good reasons why databases use indexes instead of table scans (ie, performance).
But maybe we find a route forward towards adaptive compute over the next two years. Then we can use low compute to find items of interest in the infinite contest window, and then use high compute to reason over them. Maybe this could provide a way forward on the cost issues at least.
Performance is going to remain an issue. It's not clear to me how solvable that is (sure you can imagine ways it could be parallelized but it seems likely there will be a cost penalty on planning that)
Instead of computing every token every time on the whole context, we can grab a cache to make some shortcut. We do the same in software development all the time. Of course it's a performance issue.
The only issue right now is the cost. You can make a bet that GPU performance will double every year or even 6 months according to Elon. RAG addresses cost issues today aswell by only retrieving relevant context, once LLMs get cheaper and context windows widen which they will, RAG will be easier, dare I say trivial.
I would argue RAG is important today on its own and as a grounding, no pun intended, for agent workflows.
We cannot simply state that at some point in time RAG will not be necessary. Like everything in the computer science world it always will depend on our data size and the resource constraints we have.
Unless of course we can process a corpus the size of the whole internet in <1 second. However, I doubt this can be achieved in the next 20 years.
RAG can augment the LLM with specific knowledge, which may make it more likely to give factually correct answers in those domains, but is mostly orthogonal to the hallucination problem (except to the extent that LLM's hallucinate when asked questions on a subject they don't know).
It is "search and summarize." It is not "glean new conclusions." That being said, "search and summarize" is probably good for 80%.
LoRA is an improvement, but I have seen benchmarks showing that it struggles to make as deep inferences as regular training does.
There isn't a one-size fits all... Yet.
“Stuffing relevant data into the context window rather than relying purely on training” is a solution to confabulation, though, just like providing relevant reference information to a person who is being pressured to answer a question is.
https://www.aryn.ai/post/rag-is-a-band-aid-we-need-llm-power...
We do see a world where LLMs are used to answer questions (Luna), but it’s a more complex compound AI system that references a corpus (knowledge source), and uses LLMs to process that data.
The discussion around context sizes is a red herring. They can’t grow as fast the demand for data.
https://myscale.com/blog/prompt-engineering-vs-finetuning-vs...
With RAG tools that exist today, we can already do things like
- providing summaries
- hierarchical summarization
- generation of questions / more prompts to nudge the model
- caching
- using knowledge graphs, function calling, or database connectors for non-semantic data querying
etc.
Inefficiency (in other words, higher expense)is sometimes even easier to perceive for decision-makers
The solution is finding a way for models to recognise the absence of knowledge.
1. Training a LLM is expensive.
2. Due to the cost to train, it’s hard to update a LLM with latest information.
3. Observability is lacking. When you ask a LLM a question, it’s not obvious how the LLM arrived at its answer.
There’s a different approach: Retrieval-Augmented Generation (RAG). Instead of asking LLM to generate an answer immediately, frameworks like LlamaIndex:
1. retrieves information from your data sources first,
2. adds it to your question as context, and
3. asks the LLM to answer based on the enriched prompt.
RAG overcomes all three weaknesses of the fine-tuning approach:
1. There’s no training involved, so it’s cheap.
2. Data is fetched only when you ask for them, so it’s always up to date.
3. The framework can show you the retrieved documents, so it’s more trustworthy.
The models aren't actually capable of taking into account everything in their context window with industrial yields.
These are stochastic processes, not in the "stochastic parrot" sense, but in the sense of "you are manufacturing emissions and have some measurable rate of success." Like a condom factory.
When you reduce the amount of information you inject, you both decrease cost and improve yield.
"RAG" is application specific methods of estimating which information to admit to the context window. In other words, we use domain knowledge and labor to reduce computational load.
When to do that is a matter of economy.
The economics of RAG in 2024 differ from 2022, and will differ in 2026.
So the question that matters is, "given my timeframe, and current pricing, do I need RAG to deliver my application?"
The second question is, "what's an acceptable yield, and how do I measure it?"
You can't answer that for 2026, because, frankly, you don't even know what you'll be working on.
(In response to “The best solution to reducing the problem of Hallucinations is first someone telling us what their Error rates in production are.”)
If I were to make an educated guess, one response would be akin to “one cannot correct errors without measuring the errors at prediction time”. This is incorrect; measuring prediction errors is not the only way.
Some kinds of errors can be reduced or even eliminated before they are generated. Consider the space of models that are structurally incapable of making certain classes of errors.
To explain it in another way: “turning the crank on a model” isn’t the only way to reason about it.
For example, in statistical modeling, there are kinds of models that do not suffer from out of sample extrapolation.
Also, the ideas of trust regions (from RL) might inspire alternative methods worth exploring.
[1] "Expecting Short Inferential Distances" by E. Yudkowsky 2007
If LLMs are akin to a "low resolution jpeg of the internet", RAGs allow checking of facts.
we only need a vector database if we want to do semantic vector-embedding search AND our dataset is too large for memory
long answer:
RAG is just a pattern of working with LLMs, independent of particular database technologies.
Basically it means you inject some context data or domain specific data into the prompt to achieve the following:
1. Nudging the model into the right direction so it will use the "correct parts" of it's "global knowledge". We want to nudge it into the correct domain or a domain very similar to our problem. We do this to generate better answers and reduce hallucinations or to adhere to a particular style we want to achieve. This is basically just a prompt-engineering technique.
2. Achieving in-context learning by providing some context to the model. This is also a prompt engineering technique used to make the model reason about things it does not know from the training and to allow it to reason about a particular text you provide at runtime.
What I just explained is basically the "AG"-Part of RAG: Augmented Generation. It boils down to putting some external data into the context.
The R (Retrieval), on the other hand, is about finding the correct data to put in the context.
If you copy an email and paste into the prompt of ChatGPT, you are basically already doing RAG. The R is copy pasting the Email from Gmail, and the AG is putting it into the Prompt.
So, no, generally speaking you don't need a vector Database.
In the real world you will probably have a large context (some files on the disk, a PDF file, etc.) and you don't want to manually select the relevant bits of information, of course. We want to automate the Retrieval part of RAG. This is mostly what people mean when they talk about "RAG". Essentially it is a information retrieval, or search problem.
A lot of programmers and projects which come from the LLM community will use RAG with a semantic vector search. Semantic vector search has shown to be quite good at selecting relevant context chunks from full text and it also feels natural because it will use embedding models, which are very similar to LLMs. By using vector-embedding search we simulate with a manual step what we think will happen when we put all the data into the LLM context: Selection through semantic interpretation.
In this case we need some form of "vector search", but we do not need a database for that per se, we can also do that in-memory. However, if our dataset is too large to be efficiently processed in-memory, we would need a vector database.
However, we can also use any other means of information-retrieval techniques or search techniques, like keyword search, full-text search, knowledge graphs, sql queries, and many more, to retrieve the relevant bits of information we want to pass to the model.
In that case we also don't need a vector database. Maybe some other database, maybe a search tool like elastic.io, maybe no database at all.
Supposing I do use text-embedding-ada-002 model and store the index in a vector database, will I be able use these for RAG with other LLMs such as Claude Haiku etc. ? Or does each LLM have its own text embedding model ?
The Embedding model is independent of the actual LLM used in generation. You can use any embedding model with any LLM. In fact ada is only one option and there are lot of really good embedding models, also ones you can easily run local, or ones which are a lot of better than ada, readily available. ada is not the best choice in all cases.
You need to use the same embedding for indexing and retrieval. Beyond that, you want to select an appropriate embedding; optimized for indexing the kind of content you have.
You’ve explained it in a way so that each part of the abbreviation makes sense now.
Thanks!