Comment by politelemon | Hacker News Reader

Comment by politelemon | Hacker News Reader

Parent Full thread

Where can I learn about effectively using Vector stores? I understand RAG itself, been through a few tutorials and the workflow. But I mean using a Vector store isn't just a matter of generate some embeddings, slap it in a storage, and then start doing searches like cosine similarity, hoping for the best.

Are there any guides on effectively using vector stores, and effectively searching for the use case you're dealing with? I don't even know what the various search types mean, just that everyone seems to use cosine similarity.

Hope that's making sense... my question is analogous to learning to become familiar with SQL and indexes and CTEs, while it's a common thing to implement a three-tier application talking to a database.

I feel that RAG might become a somewhat common feature of a team's tech stack alongside operational datastores (eg Postgres, MSSQL), because it is a low overhead to getting started and it persists, so it's cheaper. I just don't want to go at it blindly.

> But I mean using a Vector store isn't just a matter of generate some embeddings, slap it in a storage, and then start doing searches like cosine similarity, hoping for the best.

That is 90% of the work with RAG. RAG requires a surprising amount of QA to ensure "the best" is sufficient for your use case.

For clarification, cosine similarity is a metric not a search type, and people like it since a) it allows for more defined heuristics since cosine similarity is limited to [-1, 1] and b) it's computationally efficient (dot product, as with Euclidian distance) if the vectors are unit-normalized beforehand.

There are indeed multiple search types but HNSW is the best 99% of the time for both performance and latency, and is implemented in almost every modern major vector store.

> but HNSW is the best 99% of the time for both performance and latency, and is implemented in almost every modern major vector store.

In my experience, HNSW indexes are very expensive to build, relative to indexes like IVF. They also have a larger memory footprint. IVF, on the other hand, is pretty trivial to parallelize across multiple machines, and while I'm aware there are techniques for doing that with HNSW, I don't know the details well enough.

Also, if you review papers like "SOAR: Improved Quantization for Approximate Nearest Neighbor Search", they hint at some of the throughput barriers faced by graph-based methods like HNSW.

> That is 90% of the work with RAG. RAG requires a surprising amount of QA to ensure "the best" is sufficient for your use case.

Thanks that's insightful (and also adds to my current impostor syndrome), and makes a lot of sense... so it sounds like it's easy to write the query, but knowing the metric value to use is the testing part.

I haven't heard of HNSW but I'll give it a try. It seems PGVector has added HNSW: https://github.com/pgvector/pgvector#hnsw

Has anyone systematically compared embedding based retrieval against traditional full text search? With all of the focus on vector stores lately, it feels like the whole field of information retrieval has been nearly set aside.

Take a look at the BEIR benchmark, which has served as one of the main drivers for development of neural IR systems since its introduction in 2020.

BM25 presents a challenging cross-domain benchmark, and it wasn't till ~2022 that neural methods overtook it. If memory serves, it was the sparse neural methods like Splade, although recent dense models can also beat it.

The caveat is that BEIR is suffering from overfitting at this point.

Couple articles that are txtai-specific but general enough in terms of vector search.

Getting started with semantic search: https://medium.com/neuml/getting-started-with-semantic-searc...

txtai intro: https://medium.com/neuml/introducing-txtai-the-all-in-one-em...

Examples: https://neuml.github.io/txtai/examples/

Reply on news.ycombinator.com