Pgvector 0.4.0 Performance | Hacker News Reader

Pgvector 0.4.0 Performance

supabase.com

11 pointskiwicopple8 comments

Hey, qdrant ceo here. Paul, you and your team are building amazing open-source tech! And, of course, you can scale (almost) anything with just more hardware. As Nirant mentioned in his Twitter response. https://twitter.com/NirantK/status/1679525251213066245 So, it is not only a question of performance but also efficiency.

PS: What about offering Qdrant as an alternative to PG on the supabase platform? Just an idea. :)

i am not Paul of course, but i just want to highlight that the idea behind blogpost was to show that if you set some settings for postgres correctly, you can achieve much better performance

ie you can notice that most of results are on the same hardware as in nirantk, and we achieved 7 times better RPS with 0.98 precision on it and 19 times better RPS with 0.91 precision on the same amount of cpu and memory

goal was to compare pgvector with pgvector rather than any other technology

> What about offering Qdrant as an alternative to PG on the supabase platform

We are "all in" on Postgres, but we could definitely find ways to support qdrant (probably a Wrapper: https://supabase.com/blog/postgres-foreign-data-wrappers-rus...)

hey hn, supabase ceo here

this is a response to a recent pgvector vs qdrant benchmark. As we mention in the post, we’re not trying to prove that pgvector is faster than Qdrant. Our goal is to show that pgvector/postgres scales predictably for the workloads that most companies need.

We scale it to ~1800 requests per second with 91% precision in this set of benchmarks. Still, it could be better. Luckily there are quite a few features and improvements planned for v0.5.0 [0]:

    - Adding HNSW: an index with better speed & precision than IVFFlat (at a higher memory cost)
    - Product quantization: better storage for IVFFLAT, improving speed and recall
    - Parallel index builds: building your IVFFLAT indexes will be much faster

[0]: pgvector v0.5: https://github.com/pgvector/pgvector/issues/27

supabase engineer here,

would be happy to share details regarding our research and outcomes if you have any questions

Had a question about the "pre-warming" technique you used - why does this help? You say it helps with RAM utilization, but why?

Does it just spin up the right number of probes up so the experiments don't include this extra cost of spinning up these probes?

(apologies if this is a dumb question - I don't know enough about vector databases to know if this is a dumb question or not!)

What does "precision@10" mean in the graphs shown in the blog?

it means the precision at K(10) or precision for first 10 documents retrieved. In other words - number of relevant documents amongst retrieved 10(k) of them

So 0.98 precision would mean that 9.8 out of 10 documents are relevant on average

Reply on news.ycombinator.com