aljun_invictus

You're right, there are many trade-offs involved. I'm considering this as well, but it's best to use a widely adopted ecosystem for external use.

I offer some features that they have partially or don't have at all (such as priority queues and label filtering). The reason I raised this question is that I hope to reuse their protocol layers to reduce the burden on people during the migration process. I believe others with similar motivations exist as well (including cloud providers offering Kafka/RabbitMQ, whose internal implementations are not necessarily Kafka/RabbitMQ).

We have currently implemented both push and pull models, along with disk storage, and we provide services internally through gRPC. Our goal now is to offer these services externally. However, one downside of message queues is the fragmentation of protocols, which results in high migration costs for users. My motivation is quite simple: to leverage existing ecosystems while providing a better implementation for users and reducing migration costs (similar to how Redpanda partially implements the Kafka protocol). Currently, our implementation is in C++ and Rust, but the protocol layer can be in other languages. We have an advantage in throughput and have also implemented features like priority queues that Kafka doesn't have.

The AMQP protocol is quite complete (and allows for some customization), but I am not very familiar with the details of the Kafka protocol. I've heard it changes frequently, and I'm concerned that this might affect the implementation.

We support both push and pull models (it seems both RabbitMQ and Kafka support both). Our implementation is actually more like Kafka (for example, it supports replay), but implementing RabbitMQ's routing isn't very complicated either. However, I'm concerned about the Kafka protocol, as it seems to have been revised many times historically, which could lead to client compatibility issues. My users typically want to decouple processing logic (not for large-scale data scenarios like logging, more like RabbitMQ use cases), so I'm quite conflicted.

My new queue has better capabilities and performance, and I want the user migration to be smoother. Therefore, I hope to use a better ecosystem.

I asked this question in other communities, and they all expressed that RabbitMQ (AMQP) has more features. Can you tell me which features of Kafka you think are more complex and difficult to implement?

Honestly, my service is most similar to Kafka (it can replay), but implementing the routing of RabbitMQ doesn't seem too complicated.

Really? Because I am just a Japanese lover, and I have studied it for a while, and I often hear it in songs. Maybe I don’t have a deep understanding of authentic usage. I think I can seriously think about changing my name.

the name is inspired by a Japanese word, and I'm sorry for my unfortunate name choice , I once search on Google when I start this project, but unfortunately I turn on the safe mode ...