Hacker News Reader

Comment by nl | Hacker News Reader

I've done fine tuning too, but the reasons they mention in "Why not just use RAG?" aren't very good.

People way understimate what RAG can do, even if in general people don't talk about the right things. For example LlamaIndex spends a lot of time talking about various extractors which is the easy part. The hard thing is deciding what you are actually searching for given a chat context.

RAG is a horrible hack (and the more you understand about the more it seems so!) but it does work.

I (and I'm sure everyone else) is experimenting with surgery on an LLM so it takes a vector representation of the docs directly alongside a text input so you don't have to do the lossy doc vector -> text -> LLM context -> vector thing. Not sure why no one has shipped this yet though!

View on HN

Why is RAG a horrible hack? LLMs can draw from only 2 sources of data: their parametric knowledge and the prompt. The prompt seems like a pretty good place to put new information they need to reason with.

I guess op may be envisioning an end-to-end solution that can train a model in the context of an external document store.

I.e. One day we want to be able to backprop through the database.

Search systems face equivalent problems. The hierarchy of ML retrieval systems are separately optimized (trained). Maybe this helps regularize things, but, given enough compute / complexity, it is theoretically possible to differentiate through more of the stack.

RAG is a hack for lots of reasons, but the reason I'm focused on at the moment is the pipeline.

Say you are trying to do RAG in a chat-type application. You do the following:

1) Summarize the context of chat into some text that is suitable for a search (lossy).

2) Turn this into a vector embedded in a particular vector space.

3) Use this vector to query a vector database, which returns reference to documents or document fragments (which themselves have been indexed as a lossy vector).

4) Take the text of these fragments and put them in the context of the LLM as input.

5) Modify the prompt to explain what these fragments are.

6) Then the prompt is sent to the LLM, which turns it into it's own vector representation.

An obvious improvement to this is that the VectorDB and the LLM should share an internal representation, and the VectorDB should understand this. The LLM should take this vector input as a second input alongside the text context and the LLM should combine them (in the same way you can put a text and image into a multi-modal model)

Is that lossy?

Isn't the vector representation of the text (and the ANN index itself) lossy, and the source text itself the source of truth?

You use the vector embedding only for retrieval of the actual text, which then gets injected into the prompt.

The conversion of a conversation context to vector for retrieval is lossy and often very manual.

Reply on news.ycombinator.com