I always wondered why a RAG index has to be a vector DB.
If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.
If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.
A full-text search index using BM25 or similar may actually work a lot better for many RAG applications.
I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/
That seems like a better stacking of the technologies even now
I did similar in 2019 but typically in reverse, FTS, and a dual tower model to rerank. Vector search was an additional capability but never augmented the FTS.
So vector search would reduce the space to like 10k documents and then we'd take the document ids and FTS acted as the final authority on the ranking.
The other problem is that embeddings search can miss things that a direct keyword match would have caught. If you have key terms that are specific to your corpus - product names for example - there's a risk that a vector match might not score those as highly as BM25 would have so you may miss the most relevant documents.
Finally, embeddings are much more black box and hard to debug and reason about. We have decades of experience tweaking and debugging and improving BM25-style FTS search - the whole field of "Information Retrieval". Throwing that all away in favour of weird new embedding vectors is suboptimal.
Why not have a similarity threshold? Say, if the distance is below 0.7, do not accept the search result.
What is happening is that text is being embedded into a different space, and that format is an array of floats (a point in the embedding space). When we do retrieval, we embed the query and then find other points close to that query. The reason for Vector DB is (1) to optimize for this use-case, we have many specialized data stores / indexes (redis, elastic, dolt, RDBMS) (2) often to be memory based for faster retrieval. PgVector will be interesting to watch. I personally use Qdrant
Full-text search will never be able to do some of the things that are possible in the embedding space. The most capable systems will use both techniques
To me that just sounds like OpenSearch with extra steps.
How is this different/better than a search engine?
Vector DBs are cool, you want one handy (particularly for recommender tasks). I recommend FAISS as a solid baseline all these years later. If you’re on modern x86_64 then SVS is pretty shit hot.
A search engine that only uses a vector DB is a PoC.
For folks who want to go deeper on the topic, Lars basically invented the modern “news feed”, which looks a lot like a production RAG system would [1].
It makes sense for the hype, though. As we got LLM’s we also got wayyyy better embedding models, but they’re not dependencies.