Python messaging with ActiveMQ and RabbitMQ

Yea, boto is a great library. IME, SQS seems to be a good option only when you have low throughput. I tried running a high volume of jobs through SQS and found that I needed a lot more workers (EC2 instances) to do the work because each message was so slow to pickup. That made things cost prohibitive. I'm currently working on another AWS based project. It was initially using ActiveRecord based db queues but that, of course, failed at high volume. So I replaced it with kestrel, which works with any memcached client, and that's been great. However, kestrel doesn't do topic/group subscriptions where lots of receivers can get the same message. That's where these message servers seem to be potentially valuable options.

If you've never needed asynchronous systems and messaging, perhaps you should consider yourself lucky.

It's handy when certain actions that your system performs is really slow compared to everything else. You send a message to a queue that will result in the action eventually occurring, and then you go on with your business. One example would be building a full-text index of a large file (or group of files). You don't need to hold up your user while that slow process goes, so you send it to a queue and let worker processes do their thing. If the process is not only slow, but also compute-intensive, these worker processes can be on a different machine/farm of machines.

Very useful in a distributed systems like environment. Imagine process A needs to send some data to B, but either could reside anywhere in the cluster. Each just has to register with the broker and it does the dirty work of figuring out how to route messages.