This is key. Sure you do not need a specialized db to iterate over <10M vectors at a time, as the results show this is the order at which the latency (for this implementation) starts getting too big for a real time system: O(100ms). You can get clever with sharding and such to scale out - calculating cosine similarity and finding the top K is classic mapreduce - but that's just spreading the compute over more computers i.e. for >100M 256 dimension vector like the author chose, you'd be at several seconds of CPU time for each similarity calculation.
I don't think the vector database companies are targeting, at least with the expectation of making money now (maybe later if some grow a lot), people with that few embeddings. They are targeting use cases with WAY more vectors - think about how many vectors chatgpt must be generating for however many conversations it's producing just today, and if the technology continues to improve and use cases grow, how many other business may be generating even more embeddings. In the same way as it doesn't make sense to use Spark on a 10000 cell csv, or to buy a Tank to haul chips out of a grocer store, it doesn't make sense to use a dedicated database under a certain scale.
> Additionally, you get to save yourself the complication of standing up a vector database and waiting the ~100 seconds to index those million embeddings.
Isn't that a one time cost with the subsequent upserts amortized? Or they could even be done outside the hotpath in batches: if the db is just computing similarities to stored vectors, you could add a ton of new vectors to a new db instance/replica and then swap them.
I'm sorry if I'm being overly critical, but to me this seems like exactly what happens any time someone releases a product or technology that improves performance for really big workloads (so, almost every big data system or task-specialized db), and everybody says the technology is stupid because they don't need it at their startup with 1% of the data the technology begins to be useful for. Sure, but others have 100x your data and would benefit from it.
Let me just point out one example for why 10M vectors is actually quite small for some use cases: English wikipedia has 6M articles and it would take much more than one 256 dim vector to accurately encode some of those pages. Even with a single such vector per article you're at 0.3 cpu-s for each nearest-K calculation on an arbitrary page. And that's one admittedly large portion of an amdmittedly large website, but still a fraction of the datasize you'd get from crawling the web, looking at discord/fb/instagram/snapchat messages, reddit comments, etc.