Hacker News Reader

Comment by spencerchubb | Hacker News Reader

I believe RAG is a temporary hack until we figure out virtually infinite context.

I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on.

RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimize loss, it is just a similarity score.

Obligatory note that I could be wrong and it's just my armchair opinion

View on HN

It's not just about context length, it's about performance.

A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things.

As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.

There's also the focus. A long context means the LLM is more easily distracted. Context "gets the right neurons firing" and why RAG does well. It's also why reranking with RAG improves output, in further increases the quality and/or relevance into a smaller context.

Which is all to say really, data quality is probably the most distinguishing factor in LLM/RAG systems... to get back to OP's questions

Good point! If datasets get that large that is an argument in favor of RAG

This doesn’t consider compute cost; the RAG model is much more efficient compared to infinite context length.

Agreed. I think that RAG implemented via tool-calling, with multiple agents talking to each other, is a much much more likely evolution in the future versus a single unified model.

I could very well be wrong! But we wouldn't want LLMs to be performing lots of arithmetic calculations via exploiting hidden parts of themselves that do linear regression or whatever, far better to just give them the calculator and get results faster and cheaper. Similarly, we can give them a search engine (RAG) and let them figure it out more efficiently.

https://arxiv.org/html/2404.07143v1

“This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation”

> I believe RAG is a temporary hack until we figure out virtually infinite context.

I'd assume "large enough" context is the actual goal here, not "virtually infinite".

I think that "next level" would essentially be a RAG referential information system that gets data via search engines or databases. Maybe we will have the "google search" equivalent completely intended for LLM clients where all data is stored, searched, and returned via vector embeddings, but it could tap on exabytes of information.

That is actually already what is happening, tools like llama_index are trying to provide all the necessary integrations.

As a concept infinite context will not be discovered. We can spin up huge models with large contexts that cost more but even 1M context is 1 expensive and 2 not 3trillion(size of some datasets)

Rag is not bad attention, rag is the user not knowing the context to give the LLM

That's going to be quite a while then because LLMs don't even perform well at anything close to the current context limit. They will miss important details when the context gets over a couple thousand tokens.

Reply on news.ycombinator.com