> If data loss is unacceptable then Kafka is basically the only open-source solution that is known for not losing data
What about RabbitMQ?
What about RabbitMQ?
We reconsidered using it again for a synchronous RPC communication as we were replacing gRPC, but ended up going with nats.io instead. It does have less fearures but we are able to squeze much more juice on a smaller stack.
The biggest issue for us was however was that there is no middle server that could handle an route connections to available workers. We were using haproxy which worked ok but far from great. It was very hard to figure out how many servers need to run at any given point and thus a ton of our requests were ending up with UNAVAILABLE response.
Essentially what we needed is a synchronous RPC over PubSub which gRPC doesn't offer.
You would need to run RMQ in HA mode with multiple brokers to have any chance of not losing data.
https://www.rabbitmq.com/confirms.html#publisher-confirms-la...
To fully guarantee that a message won't be lost, it looks like you need to declare the queue as durable + to mark your message as persistent + use publisher confirm. And it looks like this costs you several hundreds milliseconds of latency.