Vector database built for scalable similarity search
milvus.io
milvus.io
If anyone from AWS/Google/Azure is listening, please add pgvector [1] into your managed Postgres offerings!
That's why we built attribute filtering into Milvus via a Mongo-esque interface. No SQL and not as performant as an RDBMS, but it's an option: https://milvus.io/docs/hybridsearch.md
That said, it's up to the individual company to decide if the added cost is worth it. Just because the cost exists doesn't mean it isn't worth it.
If your guy couldn't get a single open source software straight, you had the wrong guy :(
I can only see managed service useful when I had 100X traffic and when strong SLA is required.
https://supabase.com/docs/guides/database/extensions/pgvecto...
If the vectors are in the same database as the tabular/structured data then text to sql applications of llm's are so much more powerful. The generative models will then be able to form complex queries to find similarity as well as perform aggregation, filtering and joining across datasets. To do this today with a separate dedicated vector db is quite painful.
1) Embedding crash course https://developers.google.com/machine-learning/crash-course/...
2) What is a vector database? https://zilliz.com/learn/what-is-vector-database
3) Introduction to vector similarity search https://zilliz.com/blog/vector-similarity-search
4) ANN benchmarks http://ann-benchmarks.com
5) Unstructured data ETL https://github.com/towhee-io/towhee
Also, TBH, it is a lot cheaper to run a simple faiss index.
A good intro to the field with progression towards a full Milvus implementation could be starting with towhee[0] (which is also supported by Milvus).
towhee has an example to do exactly what you want with CLIP[1]. For icing on the cake the example notebook includes deployment of the model with Nvidia Triton Server[2], which IMO is hands down the best way to actually deploy an ML model[3].
[0] - https://towhee.io/
[1] - https://github.com/towhee-io/examples/tree/main/image/text_i...
[2] - https://github.com/triton-inference-server/server
[3] - https://news.ycombinator.com/item?id=35199418#35200266
I mean look at their system diagram https://milvus.io/static/0bc2e74d0a1b20bbfb91bdbd03f77e5e/bb.... Not fun...
1. The Node.js client is designed to be just a thin wrapper around Redis commands. The client's docs basically just point you straight at the Redis docs.
2. The `@redis/search` API is slightly different than the FS.SEARCH Redis command's api. The difference is not documented and cost me ~20 minutes. This is a pretty big problem given item 1.
3. The `@redis/search` TypeScript types didn't allow some valid FS.SEARCH calls.
2. The `@redis/search` package extends `@redis/client` with support for all RediSearch commands (you should get autocomplete for all RediSearch commands as well). If you want to get the "whole in one" package (support for redis vanilla + all redis modules) you can use the `redis` package instead.
3. Can you please share the command you are trying to run?
BTW, these examples might help:
1. https://github.com/redis/node-redis/blob/master/examples/sea...
2. https://github.com/redis/node-redis/blob/master/examples/sea...
3. Checkout `redis-om` https://github.com/redis/redis-om-node/tree/main
That being said, many of these DBs are overcomplicated for most use cases. Redis HNSW will work for many use cases.
More to come. https://github.com/redisventures to keep up with our team
From what I can tell, its competitors generally don't do this, and you have to handle generating the embeddings and all the back and forth with the embeddings yourself, setting the barrier for integration much higher.
If you are at a level where you want to get a lot more control about embeddings, I agree that there it will often be better to just use the vector search capabilities that your existing solution like Redis/Postgres are getting.
I've been playing with this extension: https://github.com/asg017/sqlite-vss
Milvus – An Open-Source Vector Similarity Search Engine - https://news.ycombinator.com/item?id=22012300 - Jan 2020 (26 comments)
(for curiosity purposes only – reposts are ok after a year or so: https://news.ycombinator.com/newsfaq.html)
I was surprised to see Elastic actually has ok support for some of this stuff, though it appears slower for most of the tasks.
So you can combine attribute-based filters along with nearest-neighbor search.
Put together this semantic search + filtering demo just last week: https://github.com/typesense/typesense-instantsearch-semanti...
And what about ScaNN and GPU index? the flexibility to support multi index type is a actually a big plus
Are they guaranteed to return the most optimal similar neighbors or will they be sloppy in return for efficiency?
The most commonly used one I've seen is HNSW, which greatly speeds up search at the expensive of some extra memory consumption. Recall is also strong - for many large-scale use cases you can get 95%+ with properly tuned parameters.
We (Pinecone) tend to attract customers who want to start, ship, and scale quickly and reliably without worrying about infra and ops overhead.
For those interested, here's a comparison with other open source vector databases: https://zilliz.com/comparison. For those who don't want to be burdened with installing and maintaining a local database, there's a managed service available as well: https://zilliz.com/cloud.
Btw, Milvus is described in your comparisons as "a fully open source and independent project", while Weaviate and Qdrant, in contrary, are "maintained by a single commercial company offering a cloud version". Why then the suggested way in the Milvus Quick Start on github is to use Zilliz Cloud?
Here's a highly simplified example: if my query was "fields related to computer science", a semantic engine could return "statistics" and "electrical engineering" while avoiding things such as "social science" and "political science".
The real focus should be to improve the recall of vector search. Pity that nobody is doing real AI research here. Money wasted in marketing and branding.
I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is.
I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are only good for vector search.
My point is I only trust stuff that focuses their own business. Especially for small startups.
The thing is to make ElasticSearch scores "comparable" to Milvus scores. Lots of ways to do this, but there's no single good solution. For example you could calculate BM25 score offline, or use TF-IDF score to do some kind of filtering. Again there's no single perfect answer. You'd have to do a lot of experiment according to your own use case and your own data to get the best results.
Also a lot of tuning needs to be done during all phases: 1) query pre-processing 2) query tokenizing 3) retrieval 4) ranking and reranking
I personally would not trust any universal "hybird-search" solutions. All toy demos.
It usually takes 5-10 good engineers to build a decent search engine/system for any real use case. It also requires a lot of turning, tricks, hand-written rules to make things work.
Others have made other suggestions, but Vespa has two unique features. First it is battle tested at a large scale, second it supports combining the keyword and vector scores in several ways. The latter is something that other hybrid systems don't do very well in my experience.
One important limitation of Milvus that I cannot ignore is that it does not support pre-filtering like Vespa and a few others do. Pre-filtering is critically important for use cases in e-commerce etc. where filters based on structured metadata are applied e.g. brand, category etc.
Try Milvus standalone instead, much simpler. I also just found their python version (https://github.com/milvus-io/embd-milvus), which is quite neat.