Hacker News Reader

I wonder how this different from HyDE. https://docs.haystack.deepset.ai/docs/hypothetical-document-...

It seems to be fundamentally the same deal except instead of passing `q` to GPT-4, they have some long-context "Memory Model" (whose details I've yet to fully understand). Also, MemoRAG uses a more conventional Retrieve/Generate pipeline downstream of the generated queries than "Contriever" (whose details I similarly haven't informed myself on).

It would be interesting to see a performance comparison, it certainly seems the most relevant one (that or an ablation of their "memory model" with the LLMs upon which they are based).

> they have some long-context "Memory Model" (whose details I've yet to fully understand)

Section 2.2 of the paper[1] goes into this in more detail. They pretrain the draft model using the redpajama dataset, followed by a supervised fine-tuning step. The training objective "aims to maximize the generation probability of the next token given the KV cache of the previous memory tokens".

This suggests that any model with long context and good retrieval performance could do the same job (and maybe better in the case of the SOTA frontier models).

[1] https://arxiv.org/pdf/2409.05591

Reply on news.ycombinator.com