Hacker News Reader

Comment by ritwikt | Hacker News Reader

Would be nice to know why RabbitMQ/ZeroMQ didn't fit the bill [scale/speed?] -- I can somewhat understand the not relying on SQS but then there is anyways a reliance on EC2 right ?

View on HN

I thought the same thing about Rabbit. PinLater's ACK system feels like a questionable reinvention of AMQP. I'm probably missing something.

Rabbit's advertised throughput doesn't reach 100,000/sec, but you should be able to shard your way around that. Or use Apache Kafka, which has the sharding built in.

Just for comparison, LinkedIn is doing about 2.5 million messages per second (on average per day) with Apache Kafka.

http://www.slideshare.net/ToddPalino/enterprise-kafka-kafka-...

Precisely, this is the same query I had when I glanced down the post. It will be great if the author or someone from the team respond to the various trade-offs and why they did not go ahead with Kafka ?

(I am the author of the blog post). I posted a reply below earlier this morning that hopefully addresses your question.

Looks like they needed to have built-in support for time based scheduled execution of jobs at a specific time in the future. Rabbit MQ does not have that (that I'm aware of) without doing some horrible hacks around one queue per message.

ActiveMQ's had that for a couple of versions though. Another highly performant/available, atomic, consistent, durable and mature queueing server.

Correct. Celery's implementation dequeues but doesn't acknowledge the task, holding it until the timer is reached.

I'm curious here as well.

Reply on news.ycombinator.com