Hacker News Reader

Comment by jszymborski | Hacker News Reader

Ok, I think I get it now from scanning the paper and reading Eq. 1 and 2.

Normally RAG just sends your query `q` to a information retrieval function which searches a database of documents using full-text search or vector search. Those documents are then passed to a generative model along with your query to give you your final answer.

MemoRAG instead immediately passes `q` to a generative model to generate some uninformed response `y`. `y` is then passed to the information retrieval function. Then, just like vanilla RAG, `q` and the retrieved documents are sent to a generative model to give you your final answer.

Not sure how this is any more "memory-based" than regular RAG, but it seems interesting.

Def check out the pre-print, especially eq. 1 and 2. https://arxiv.org/abs/2409.05591

EDIT: The "memory" part comes from the first generative model being able to handle larger context, covered in Section 2.1

View on HN

  Not sure how this is any more "memory-based" than regular RAG, but it seems interesting.

I can't remember where I read this joke, but as a self-proclaimed Cognitive Engineer I think about it every day: "An AI startup's financial evaluation is directly proportional to how many times they can cram 'mind' into their pitch deck!"

thanks for boiling it down to the most salient point... to me, their approach is just query rewriting, which is pretty standard when doing RAG.

There's a lot there about the generative model ("Memory Models") in the paper, so perhaps I've misrepresented it, but generally speaking yah I agree with you. It doesn't sound like a fundamental change to how we think about RAG, but it might be a nice formalization of an incremental improvement :)

Agreed. In the RAG space there are a million Open Source projects on GitHub all calling it memory, recreating the same thing over and over again.

Not exactly, they use a small but long-context model that has the whole dataset in its context (or a large part of it) to generate the chunks as elements of the reply, before passing those to the final model. So the retrieval itself is different, there is no embeddeding model or vector db.

I do memory-based RAGs using Semantic Kernel function calls which do specialized memory caches with useful calculations based on telemetry data. It is so simple to do and I love the LLM figures out how to call the SemanticFunctions on its own.

ollama and langchain can do something simimlar.

I wonder how this different from HyDE. https://docs.haystack.deepset.ai/docs/hypothetical-document-...

It seems to be fundamentally the same deal except instead of passing `q` to GPT-4, they have some long-context "Memory Model" (whose details I've yet to fully understand). Also, MemoRAG uses a more conventional Retrieve/Generate pipeline downstream of the generated queries than "Contriever" (whose details I similarly haven't informed myself on).

It would be interesting to see a performance comparison, it certainly seems the most relevant one (that or an ablation of their "memory model" with the LLMs upon which they are based).

> they have some long-context "Memory Model" (whose details I've yet to fully understand)

Section 2.2 of the paper[1] goes into this in more detail. They pretrain the draft model using the redpajama dataset, followed by a supervised fine-tuning step. The training objective "aims to maximize the generation probability of the next token given the KV cache of the previous memory tokens".

This suggests that any model with long context and good retrieval performance could do the same job (and maybe better in the case of the SOTA frontier models).

[1] https://arxiv.org/pdf/2409.05591

Reply on news.ycombinator.com