Hacker News Reader

I've actually been looking at Milvus for storing embeddings, why do y'all have options for both Pulsar and Kafka inside Milvus? Seems like unnecessary choices for what we hoped would just be a plug and play vector database

View on HN

The design of Milvus 2.x follows this paper we published a while back: https://arxiv.org/abs/2206.13843. In short, we used Pulsar to implement the write-ahead log, which provides coordination and a single source of truth across all Milvus components.

You're right in that it's a bit heavyweight, so we're working to see how we can make pub/sub and other cluster components lighter and more efficient overall.

Based on some superficial research yesterday, Weaviate looks like an easier option for messing around locally, but Milvus looks better for a production use case.

Reply on news.ycombinator.com