Pgvector: Fewer Dimensions Are Better | Hacker News Reader

Pgvector: Fewer Dimensions Are Better

supabase.com

66 pointstodsacerdoti12 comments

we were lucky to discover the new gte-small[0] model while writing this, which is so new that it doesn't have a lot of documentation around it. The neat thing about this model is that it out-performs OpenAI's `text-embedding-ada-002` on Hugging Face's MTEB Leaderboard[1] and it's only 384 dimensions. This is great for Postgres & pgvector - it means less space and faster queries.

There are some trade-offs (it was trained on English-only text), but this is a good indicator of the direction where embeddings might go (code-specific, language-specific, etc)

[0] https://huggingface.co/Supabase/gte-small

[1] MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard

It seems worth noting that the input of gte-small is limited to 512 tokens. The tokenizers for sure aren’t the same, but I imagine this is significantly less than ada-002, whose input is limited to 8191 tokens. That said, I don’t imagine that embedding huge full documents is necessarily the right approach.

I would love to see a comparison for some typical use cases using various methods of chunking input documents.

should be yes, but even in examples from openai, they usually do splitting into chunks

For example in chatgpt-retrieval-plugin[0] repo default chunk size is just 200 tokens

this is anyway a limitation, no doubt, but chunking is pretty often used

[0] https://github.com/openai/chatgpt-retrieval-plugin/blob/main...

OpenAI's embeddings don't add that much. For similarity search you can use a number of open source models from huggingface with small embeddings sizes and do as well or better than OpenAI’s embeddings.

agreed, and this is the other nice thing. You can use any embedding system for search and retrieval, and then use OpenAI/GPT-3.5/GPT4 for generative use cases (which is still miles ahead for many functions)

fwiw, 98% of Supabase project use OpenAI for embeddings, so a lot of this content is just to help devs identify the best options they have available

I'm guessing most people use OpenAI embeddings because it's very convenient to use an API instead of running the model yourself, and they perform well enough that people aren't actively looking for other models.

Oh - that's a nice model. Can probably run with 0 issues on your low-end linux CPU machine.

> and it's only 384 dimensions

And the model is only 70MB!

heya, supabase engineer here, i coauthored the part about benchmarking different models with pgvector to compare QPS when using fewer dims models.

btw, you can find the dataset with embeddings generated by all 3 mentioned: text-embedding-ada-002, all-MiniLM-L6-v2, and GTE-small on huggingface[0]

and big thanks to Stephan Sturges for his dataset[1]. we just extended his OpenAI ones and texts themselves with oss ones

[0] https://huggingface.co/datasets/Supabase/wikipedia-en-embedd...

[1] https://www.kaggle.com/datasets/stephanst/wikipedia-simple-o...

Kismet this is published today. We've just announced Zep's[0] new Document Vector Store[1] built on pgvector, and the embedding model we ship with is all-MiniLM-L6-v2. Very fast, low-memory, and surprisingly performant.

Our focus with Zep is on LLM app use cases, and in particular, searching over chat histories and documents for RAG apps. You can, however, use Zep to turn any Postgres instance into a vector store with great developer experience. pgvector index configuration and query tuning can be challenging. We've tried to do much of this work for developers.

Also, I'm super excited about the prospect of HNSW support in pgvector, slated for pgvector 0.5.[2]

[0] https://github.com/getzep/zep

[1] https://blog.getzep.com/introducing-the-zep-document-vector-...

[2] https://github.com/pgvector/pgvector/issues/181#issuecomment...

So, better performance at a lower cost. But what about quality?

the gte-small model actually outperforms OpenAI's `text-embedding-ada-002` on the MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard

The highest ranking is `gte-large`, which has 1024 dimensions - larger than `gte-small`(384), but still smaller than `text-embedding-ada-002` (1536)

Reply on news.ycombinator.com