Hacker News Reader

Comment by virmundi | Hacker News Reader

RabbitMQ. It just works. We're operating with in the normal use case for it. We turn it on, transfer 50k or so messages a day, then forget about. The memory is fairly low, CPU is low and constant. The connections for month or year long clients stay open.

View on HN

I'm actually a fan of rabbitMQ generally, but I'm not sure I'd put it in the category of notoriously reliable software. It can throw away acknowledged writes (afaik https://aphyr.com/posts/315-jepsen-rabbitmq has still not been fully fixed).

50k messages/day is pretty low throughout and under that load it should plod through just fine.

Under much higher loads than that (10's - 100's millions of messages day) I've seen it die in a pretty ungraceful manner. At that scale, Kafka is a good candidate.

Under those high loads I've seen every single MQ fall over like a house of cards. At lighter loads you can get the job done with other tools that scale but with a little bit more pain. Leaves me super skeptical about them on the whole.

We push on the order of ~a billion messages / day into one of our Kafka clusters without having too much scaling difficulty.

Indeed. We have completely replaced all queues with Kafka. In fact, I'm not sure what it would take for us to even try anything else at this point.

We process at least 30 million messages a day with sidekiq and had to start sharding message queues for different purposes. At some point it's not worth trying to optimize the messaging.

In my experience, RabbitMQ is close to the opposite of reliable.

When I think of reliability, I think of Postgres: it works well, and it's hard to make it not work reliably. With RabbitMQ, it's hard to make it work reliably.

We had to use a message queueing system for school, they recommended some Oracle thing. Other systems were allowed though, and most of the people chose RabbitMQ.

Can confirm, it's extremely easy to make Python, Java and PHP talk to each other. I haven't found anything it can't do yet, and it's not hogging resources either.

We poked RabbitMQ, Gearman, SQS, a few others, and never got the reliability we demanded. To be fair, we're handling in the order of 50 million messages a day.

Ended up just writing a simple file system based queue in Golang and it's working spectacularly. Just let the SSD do the work. Couple hundred loc all told. We rarely crack 10-15% cpu and memory use sits statically at around 30mb.

I've gone through a few things too, eventually settled on Redis.

Would be interested in seeing your queue, if it were public?

I'd love to make it public, might be a while with company politics being what they are. We have a bad habit of not even sharing with other parts of the company.

50K is nothing to talk about, really. Over a workday, that's like 2 a second. Even at 5x that much, it's enough for many uncached random hard disk IOs per message.

Reply on news.ycombinator.com