Do we think about vector storage wrong?
hachyderm.io
hachyderm.io
Vector search isn't all that new a concept. For example, the annoy library (https://github.com/spotify/annoy) has been around since 2014. It was one of the first open source approximate nearest neighbor libraries. Recommendations have always been a good use case for vector similarity.
Recommendations are a natural extension of search and transformers models made building the vectors for natural language possible. To prove the worth of vector search over keyword search, the focus was always on showing how the top N matches include results not possible with keyword search.
In 2023, there has been a shift towards acknowledging keyword search also has value and that a combination of vector + keyword search (aka hybrid search) operates in the sweet spot. Once again this is validated through the same benchmarks which focus on the top 10.
On top of all this, there is also the reality that the vector database space is very crowded and some want to use their performance benchmarks for marketing.
Disclaimer: I am the author of txtai (https://github.com/neuml/txtai), an open source embeddings database
Most older IR textbooks promote the idea of viewing keyword search as a maximization problem of the weighted bit vector product between the query and the document, where the weight matrix is something like BM-25.
The thing about vector search today is that it is an absolute gold rush, Pinecone had the right idea but the wrong product (sorry SAAS is toxic), many of the latecomers are… late, and will find the whole startup and VC process will cost precious months when customers want to build products right now.
Note for that patent search engine we had a quite a different idea than is fashionable today, that is, like very old TREC we care about the quality of results you got 1000 or 2000 results in in the assumption that a patent researcher wanted to be comprehensive. When I was first into IR it was in the context of “how is Google so much better than other search engines?” and one part of it was that based on the way search engines were being evaluated at TREC, Google was not a better search engine* because it was so focused on the first page.
It has come full circle now, I have talked to people recently who complain that current TREC evaluations are too focused on the first few results.
Given that vector and hybrid search are used in many situations at scale, I think to make the kind of claims you're trying to make, it might help to quantify exactly when you think it will be too slow or what experiences you have had of struggling to make it fast enough.
In terms of performance, vector search is definitely slow but it's come a long way. There are a number of solid 384 dimension models that work well.
When using vector search, you're making two leaps - that your document can really be summarized in a vector (keeping its important distinguishing semantic features), and that it's possible to retrieve similar documents using high-dimensional vectors. Both of these are lossy and have all sorts of caveats.
What would be great is if these new vector DB companies show examples where the top-ten "search engine results page" based on semantic embedding / vector search is clearly better than a conventionally-tuned, keyword-based system like ElasticSearch. I haven't see that, but I have seen tons of numbers about precision/recall/etc. What's the point of taking 10ns to retrieve good vector results if its unclear if they're solving a real problem?
It's funny seeing this come up now as this was my full time job ten years ago and even then most of the research had been completed decades prior.
For example, ANN indexing with an index like HNSW scales pretty much logarithmically. A bit oversimplified, but roughly we see search times double with every 10x of the search space. In addition, calculations like euclidean distance and dot product (and therefore cosine sim) can be parallelized very well at a hardware level, e.g. AVX2/AVX512/neon and similar SIMD techniques.
With BM25, this isn't quite as simple. We have algorithms such as WAND and Block-Max-WAND, which help eliminate _scoring_ elements that cannot reach a top-k spot. However, the distribution of terms plays a big role here. As a rule of thumb, the rarer a word is the fewer documents require scoring. Let's say you have 1B documents, and a 2-term query, but each term matches only 100k documents. If you AND-combine those queries, you will have at most 100k matches, if you OR-combine them, you will have at most 200k. The fact that there were 1B documents indexed played no role. But now think of two terms that each match 500M objects. Even with the aforementioned algorithms – which rely on the relative impact of each term – there is a risk we would now have to score every single document in the database. This is, again, over-simplified, but my point is the following:
ANN latency is fairly predictable. BM25 latency depends a lot on the input query. Our monitoring shows that in production cases, when running a hybrid search, sometimes the vector query is the bottleneck, sometimes the BM25 query is.
> Hybrid search can be used to first filter a smaller number of candidates and then rerank them using vector search
My post is already getting quite long, but want to quickly comment on this two-step approach. For Weaviate, that's not the case. BM25 and vector searches happen independently, then the scores are aggregated to combine a single result set.
Yes, you could also use BM25 as a filter set first, then re-rank the results with embeddings, however you wouold lose the BM25 scores. If you want to use keywords just as filters, you can do that much, much cheaper than through BM25 scoring. In Weaviate matching keywords to create a filter set, would only require AND-ing or OR-ing a few roaring bitmaps which is orders of magnitides more efficient than BM25 scoring.
Disclaimer: associated with Weaviate.
Do you have evidence that this hybrid approach is better than a vector search? That would be surprising.
If you run a web search for "vector hybrid search" you'll come across plenty of articles for a number of vector databases discussing the topic.
BM25 is often a baseline for improving upon. But I see improvements using lots of techniques, not just vector retrieval.
And I'm sure you're aware of the BEIR paper: https://arxiv.org/abs/2306.07471. Elastic references that in this blogpost: https://www.elastic.co/blog/improving-information-retrieval-...
I agree that BM25 retrieval + vector re-ranking can work. But vector search does bring results to the table that vanilla BM25 can't, even with a large retrieval window. So I do think there is a place for both with the usual "it depends on your data/requirements" caveat.
I am not disagreeing vectors are useful, but I think benchmark based evidence is not the same as deploying a solution that must scale, be constantly updated, serve many use cases like filtering, search syntax, etc customers want.
And plus I think there’s a real danger Of herding to vector retrieval (even then one view of it) which cuts off exploration of diverse solutions.
I invested a lot of time with the last txtai release adding a minimal dependency Python-based BM25 component (https://neuml.hashnode.dev/building-an-efficient-sparse-keyw...). And keyword-only indexes are supported if one desires.
I'm with you 100% on not herding to any one way for any problem. I still remember the pre-2023 world where you weren't pressured to work LLMs into everything.
You could force the data team to go through hoops like multimodal embeddings, multiple indexes, post-filtering... or figure it out
Near-term, it makes sense to do one thing well, so many new vector DBs are really mutable vector indexes with lackluster search. But eventually, if you are not a narrowly-scoped kernel (faiss) but the main search platform needing $$$-level marketshare for growing payrolls, you're incentivized to figure search out beyond ANN, and your competitors are advertising that they are doing the same. So you add post-filtering, and then eventually, maybe even push-down/dynamic.
Source: We have been happily using on-the-fly GPU knn embeddings for years in pygraphistry, but when we added billion-scale vector indexing to power louie.ai RAG search for real-time news & database querying, we outgrow pure ann on our first user, and I'm thankful we did diligence on our DB selection
- We would have paid more for storing billion+ rows in-memory than our gov & enterprise customers were paying for a full product. Since then, ivfpq etc have gotten more popular to do responsive searches for bigger-than-memory datasets. If we needed small in-memory workloads, we could have done faiss+pandas+fastapi and moved on.
- Vendors pitched themselves as managed databases yet their users complain about them going down & losing data
- Many did not support basic search operators: time filtering, ...
- Few/none had real modern security, e.g., row-level multi-tenant ABAC
I haven't been tracking on a per-vendor basis as each publicizes making piecemeal progress on these. We found one that did #1-3 without failing us yet, and we're biting the bullet to implement #4. A scalable OSS answer to #4 might be the one that causes us to switch.
Disclaimer: I'm with Vectara (https://vectara.com), we provide an end-to-end platform for building GenAI products.
I'm surprised to see less mention of the use of vector search in RAG. The reason top 10 (or less) is so important in RAG is that you only have so much context to work with (and even in large context models, you still would prefer to minimize the total amount of context you're sending the model).
Everyone I know working seriously on vector search is using it to plug into a prompt for an LLM. For this use case top 10 is a fairly important metric.
[Insert technique name] is a good technique, but it can be further improved if you enable [some complex workflow]. [Some example of how it improved a particular case]. We at [Company] already do this at scale and you can use it out of the box if you use it. Go make better things.
Nothing against people doing this stuff. It just makes me think how difficult is it to promote open source projects today.
Regarding Vector DBs, most have advocated for weaviate and pinecone. Incidentally, all run into quantization problems at scale (1M+ vectors) if the clusters are to be adjusted, and handle it differently. i tried it for a 6000 vector product[1] and got the best results by using a pkl file rather than any of the dbs.
[1] Not sharing the product here as such. We just scraped a govt tax website to create a search engine on top of that. Helps as the data is not indexed by google, and we had no interest from customers in wanting to chat. They were interested in original source along w a summary.
Vector databases make sense when you have a lot of data or some other requirements (e.g. persistence, online updates, complex filtering, etc) that you would rather not implement yourself.
Another way to look at 700k pages is 2333 300 page books.
Build protocols, not products! I tell people.
It's a terrible advice actually, unless you like eating noodles and scraping by. But it's still the right way.
We're still waiting for the antirez' Redis version of Vector DBs. Maybe he should do one! :)
I've been using it for a side project to do semantic search on books[3] and have been really happy with its performance. (Not affiliated with any of this, was mostly interested in exploring existing well-performing, fairly standard tools with low latency)
[1] https://redis.io/docs/interact/search-and-query/search/vecto... [2] https://ann-benchmarks.com/#redisearch [3] https://viberary.pizza/how
Can you elaborate? We (at Pinecone) have dozens of customers with over a billion embeddings, still serving queries in under 300ms (p95) with index updates in <1s. If you saw performance issues with just 1M embeddings then something else is going on that we may be able to help with.
I was just testing to see if there is a DB which does not run into such problems, before putting a business case. I am not discouraging the use, just saying that most cases do not need a vector DB. The reason we did not choose one was because our business case's number of vectors was small.
Maybe your question is referring to the need to have quantization at larger index sizes? In which case, yes would typically be true because you're wanting to either (1) minimize the index size when quantizing it, or (2) optimizing the query space (to search through less). Whether you want (1) or (2) will impact on the type of quantization being performed (basically 1 == product quantization, and 2 == inverted index)
So once you get to the 1M+ size, you need to consider quantization in some form - or you can go with graph-based retrieval, if you don't mind using a lot of disk space.
Amazingly, a true semantic search would work well. And to do semantic search in scale, you'll need a vector database.
Word!
We have around 2 million 768-length embeddings + metadata -- writing too fast to Weaviate quickly results in OOMs. We circumvented this by very slowly trickling our data through to Weaviate, but this causes many-hours-long writes.
I'm experimenting with dumping our data into postgres and building an hnsw index with pgvector once all writes are done. This has reduced our time to maybe 2 hours at most (the bulk of that being index building time, of course).
1) Limit the maximum size of an update. This causes gradual degradation of the index, which is ordinarily dealt with through periodic consolidation — but now this this consolidation will be disabled in the main write mechanism.
2) Enable a background consolidation process which runs concurrently. At the end of the consolidation process a short write lock will need to be obtained while you replay all deletions that have occurred since consolidation started, but the time required isn’t problematic.
This fix required major surgery to the Lucy indexer core code, so it’s something that that would only be feasible if the data store itself supported it.
Disclaimer: I'm from Pinecone, hello!
1. gRPC. Using gRPC to write vectors has had a really nice performance boost. It is released in Weaviate core but here is still some work on do on the clients. Feel free to get in contact if you would like to try it out.
2. Parameter tuning. lowering `efConstruction` can speed up imports.
3. We are also working on async indexing https://github.com/weaviate/weaviate/issues/3463 which will further speed things up.
In comparison with pgvector, Weaviate has more flexible query options such as hybrid search and quantization to save memory on larger datasets.
Do you have more than a terabyte of embeddings? No? Do yourself a favor and use pg_vector with a HNSW index, it'll do just fine. Operationally, it is very hard to beat Postgres.
I've worked on a project that searched through 50 million vectors with linear brute-force search, just with a sprinkle of AVX (and it used the brute-force search, because the curse of dimensionality killed all the smarter approaches).
I think your point is right though. Searching through these is requires an index of some sort at any reasonable scale (not google scale).
Not the point of the article, but it is striking to see how mastodon is way better in terms of usability/aesthetic and general UX than Twitter or 'x'.
I've been mildly interested in mastodon for quite some time from a political perspective (remove large support of big companies where possible), but I didn't realize it was just sooo much better to use.
Not rocket ship “going to explode and burn out” power. There’s enough there to keep people engaged. The decentralized nature means there’s always corners of it alive and growing.
My mental model is that the job of vector storage is to just fetch the closest vectors to a target vector for some proximity metric, with the model (or method , more generally) that generates embeddings being its own distinct problem. You may later do things like passage re-ranking and maximal marginal relevance to refine it, but again, different task.
I understand vector search against tons of other vectors is an approximation endeavor due to practical concerns, so it doesn't seem strange for vector store accuracy to be measured in terms of top-k retrieval. Post processing and the quality of the vectors themselves is a different thing to judge.
I know, FWIW, Elasticsearch has been investing a lot in this space lately. It's not perfect, but its bound to get better.
A key consideration for the vector size limitation is that storing and working with large amounts of huge vectors gets expensive quickly. Simply storing lots of huge embeddings can take up a lot of space.
And of course using knn with huge result sets is very expensive, especially if the vectors are large. Having the ability to filter down the result set with a regular query and then ranking the candidates with a vector query helps keep searches responsive and cost low.
If you are interested in this, you might want to look at Opensearch as well. They implemented vector search independently from Elasticsearch. Their implementation supports a few additional vector storage options and querying options via native libraries. I haven't used any of that extensively but it looks interesting.
That depends... How fast do you need it to be, how many vectors will you be searching across, and how often will the index be updated? If the answers are anything resembling "very, many, and often" then you'll want to at least compare with a DB that's been purpose-built for vector search. I'm talking >10M vectors, <100ms, <hourly updates. If the workload is anything less than that, then just use whatever is most convenient -- which in your case could be Elastic.
(Disclosure: I'm from Pinecone. The above is based on independent testing.)
This looks like a good overview if you want to read about it: https://recsysml.substack.com/p/two-tower-models-for-retriev...
2. Okay, let's productize!
3. Ouch, the results aren't quite production-ready...
4. Huh, the bottleneck seems to be the search & retrieval process...
5. Oh, I guess I don't know anything about search, time to catch up on the field...
6. Ok, now I need to build an inverse index or knowledge graph and augment it with embeddings.
7. Hm, actually, language models (including BERT-based, Splade) could be pretty useful in building structure out of my dataset...
IMO the best way to incorporate transformers into search has yet to be discovered, although whatever OpenEvidence.com is doing seems to work well. If I were still in grad school I would look into some kind of PageRank / graph NN approach to the problem, or maybe hyperbolic embeddings.
Reason being that for many corpora there exists some kind of internal structure or linking that can be exploited for speed and accuracy, which you don't quite get from nearest neighbors in a Euclidean space.
I tried to provide a very clear explanation of embeddings here, since I know lots of people haven't yet learned what they are or why they are interesting: https://simonwillison.net/2023/Sep/4/llm-embeddings/
Disclaimer: I work at Algolia.
Are vector databases built on top of rd trees?
I do see lots of us in the vector DB space demonstrating on real data the speed and semantic retrieval capabilities of vector DBs. Not many of us publish traditional IR benchmark #s simply because vector search is a pretty different ballgame than traditional keyword/BM25 search (as you know).
However, since most search practitioners have developed their mental models around things like TFIDF and BM25 + Learning To Rank, many vector DBs tailor their language to fit that heuristic.
If you look at vector DBs on their own -- I like to look at them as a piece of infra to use in conjunction with pre/post processing + RAG + BM25 + whatever else you want -- you'll find that measuring their accuracy against traditional IR benchmarks is a bit like fitting a square peg into a round hole. It almost fits! But not quite.
I think one of the starkest differences between the vector search and traditional search when it comes to retrieval evaluation is the great difficulty involved in creating judgment lists to measure vector search performance. Since vector search inherently retrieves documents that are approximately conceptually similar to a query, the initial recall set, especially with an IRL dataset (e.g. Reddit, Shopify, etc.) is a moving target. Language evolves. Idiosyncrasies pop up. Etc.
Not to say that those ^ issues don't plague keyword search as well. But the powerful thing about vector databases IMO is that unlike traditional indices that need lots of fine tuning and IR experts, a vector DB will capture these new additions to your domain's vocabulary immediately, based on the surrounding context in the document.
Of course you can do similar things with traditional search engines by incorporating curated lists of synonyms, antonyms, stop words, etc. but that takes time (despite engines like Solr having built-in lists of these kinds, search engineers often have to curate them further, manually, because their product's domain is so niche) and lots and lots of query-stream analysis (I remember personally doing lots of the latter at Reddit, hehe).
TLDR: - I think pure vector search platforms should be evaluated differently than traditional keyword search platforms - I think vector search is a tool -- simply one of many -- that search engineers should use to make their search engine results more relevant.
Disclaimer: I'm from Pinecone (and hi, Doug!)
Have you tried adding whatever terms you want retrieved back as 'metadata' attached to your vectors?
This is highly accurate, most vector database companies don't talk about the shortcomings of vector representations for search.
>TLDR: - I think pure vector search platforms should be evaluated differently than >traditional keyword search platforms - I think vector search is a tool -- simply >one of many -- that search engineers should use to make their search engine >results more relevant.
This is a contradiction. One one hand you say that you want to improve relevance, on the other hand you say that vector search as a tool cannot be evaluated as other models (tools).
We have plenty of open information retrieval datasets (both full retrieval and ranking) where you can compare different methods or tools and assess the relevance impact.