Hacker News Reader

Comment by softwaredoug | Hacker News Reader

I think people assume RAG will just be a vector search problem. You take the user's input and somehow get relevant context.

It's really hard to coordinate between LLMs, a vector store, chunking the embeddings, turning user's chats into query embeddings - and other queries - etc. It's a complicated search relevance problem that's extremely multifaceted, use case, and domain specific. Just doing search from a search bar well without all this complexity is hard enough. And vector search is just one data store you might use alongside others (alongside keyword search, SQL, whatever else).

I say this to get across Postgres is actually uniquely situated to bring a multifaceted approach that's not just about vector storage / retrieval.

View on HN

Yes ... and no.

You're right for many applications. Yet, for many other applications, simply converting each document into an embedding, converting the search string into an embedding, and doing a search (e.g. cosine similarity), is all that is needed.

My first attempt at RAG was just that, and I was blown away at how effective it was. It worked primarily because each document was short enough (a few lines of text) that you didn't lose much by making an embedding for the whole document.

My next attempt failed, for the reasons you mention.

Point being: Don't be afraid to give the simple/trivial approach a try.

I'm essence what you are saying is it worked for a demo but not for an actual application

Most people need it just for a demo, though. I'm not developing any applications for it, but I have several ideas in mind for RAG just for my personal use case on my local machine.

And for the one that worked, BTW, would work almost unchanged in an application. I may put it online one day as several people have expressed interest - simply because there are no existing sites that do the job as well as my POC. The challenges will mostly be the usual web app challenges (rate limits, etc), but the actual RAG component doesn't need improvement to be the "best" one out there.

Totally agree - the "R" in RAG is about retrieval which is a complex problem and much more than just similarity between embedding vectors.

Not sure if postgres is uniquely situated, elasticsearch can do everything you mentioned and much more.

Yes es/os has been especially great to the louie.ai team. We need to deal with a lot of data - we handle talking to real-time news, logs, social media, alerts, threat feeds, etc - and these systems were previously already scaled for large document corpuses with rich metadata even before having the vector index added. Eg, imagine having to deal with hot vs cold storage for diff queries, and auto-graduating data.

When we looked at the pricing for vector DBs and less mature DB extensions, the real playing field shrunk down pretty quick. (And esp if you want managed & unified system wherever possible vs a NIH Frankenstein.)

Reply on news.ycombinator.com