Hacker News Reader

Faiss is a wonderful vector search library - in particular, the ability to do hybrid indexes e.g. IVF-PQ, IVF-SQ is great. We (https://milvus.io) use it as one of the indexing options (along with Annoy, Nmslib, and DiskANN) to power our vector database.

View on HN

I've actually been looking at Milvus for storing embeddings, why do y'all have options for both Pulsar and Kafka inside Milvus? Seems like unnecessary choices for what we hoped would just be a plug and play vector database

The design of Milvus 2.x follows this paper we published a while back: https://arxiv.org/abs/2206.13843. In short, we used Pulsar to implement the write-ahead log, which provides coordination and a single source of truth across all Milvus components.

You're right in that it's a bit heavyweight, so we're working to see how we can make pub/sub and other cluster components lighter and more efficient overall.

Based on some superficial research yesterday, Weaviate looks like an easier option for messing around locally, but Milvus looks better for a production use case.

Reply on news.ycombinator.com