Evaluating persistent, replicated message queues

It would be nice to see the strongest of the various Apache contenders represented here too. Once upon a time, there was just ActiveMQ to worry about:

http://activemq.apache.org/

It's long been very popular, and has all the flexibility and enterprisey goodness you could want, but has not had a reputation for speed. However, i believe there was a release a year or two ago that significantly improved performance. I vaguely remember this being based on an infusion of technology from the Qpid project, which itself has a broker:

http://qpid.apache.org/components/java-broker/index.html

Or maybe that was what led to Apollo, which is the faster, more reliable successor to ActiveMQ, but which hasn't, er, succeeded ActiveMQ:

http://activemq.apache.org/apollo/

Or maybe i just imagined the link to Qpid.

I'm fairly hazy on how these bits of software relate, and i can't promise you there aren't more somewhere in the vast Apache mansion. Even figuring that out would be useful, never mind the benchmarks!

I am not very familiar with messaging or message queues, so wanted to seek out some advice before spending time reinventing some well known wheel poorly.

I would like to have a broker that sits between an application and several endpoint services to which the application makes requests via the broker. These endpoint services have rules, typically throttling rules (i) you cannot submit more than x number of request per second (ii) you cannot queue up more than y number of requests at our end (iii) you must put a delay of z seconds after we have responded to a request of yours, etc., etc.

Is there a defacto standard open source software for this situation, that would take a config file describing the policies to follow for an end point. My requirement is pretty toy'ish, small to medium scale only. Just taking a look at Redis docs (am new to that too) it seems it will not be that hard to write out a simple solution. Would that be the way to go ? Or is there something out there that already handles this.

What this article fails to talk about are primarily the failure scenarios of the queues under partitions. Does the queue become write-unavailable, read-unavailable, inconsistent, or otherwise during network partition? Can the queue remain partially available during network partition?

These are the situations, in a scaling, production system that really matter.

required reading for anyone using distributed systems: http://aphyr.com/tags/Jepsen

in particular:

http://aphyr.com/posts/284-call-me-maybe-mongodb

http://aphyr.com/posts/315-call-me-maybe-rabbitmq

Would love to see Kafka represented here.