A list of queueing libraries | Hacker News Reader

A list of queueing libraries

queues.io

44 pointsstrzalek25 comments

He didn't mention IBM WebSphere MQ or Oracle Tuxedo that have being used in complex scenarios before the products in the list.

I would add my implementations and resources, even if they were for educational purposes:

i) Persisting Native Python Queues: http://blog.databigbang.com/persisting-native-python-queues/

ii) Adding Acknowledgement Semantics to a Persistent Queue: http://blog.databigbang.com/adding-acknowledgement-semantics...

iii) Esoteric Queue Scheduling Disciplines (an essay about a new kind of queue): http://blog.databigbang.com/esoteric-queue-scheduling-discip...

iv) Using Queues in Web Crawling and Analysis Infrastructure (just informative): http://blog.databigbang.com/using-queues-in-web-crawling-and...

No Celery, no RabbitMQ, but 0mq, i don't know if i should take this seriously..

Just putting some google search hits on a dedicated website, aren't we?

I've always found having to deal with background workers a pain in the butt. Not only do you have to set up additional servers (queue and worker servers), but also you run into problems as a result of your own naivety--such as not using a connection pool for workers so the startup time is too slow, or using a library that doesn't handle timeouts. Also, I find that I have little visibility into the queuing throughput, or if I chain jobs for workers (result of one worker goes to another worker), I have no idea if the data made it all the way through.

Does anyone else find it a pain, or just me?

Also, has anyone ever tried workers as a service, like iron.io? Are there others, and was it worth it?

Can you give more information about the context where you are using background workers and queues?

Amazon Simple Workflow might help: http://aws.amazon.com/swf/

Personally, I use queue_classic + postgres triggers to keep track of each of the job history. So I have a history of all the jobs, when they started, when they finished, the data in each job, etc.

But yes, it's a pain.

http://torquebox.org/ is supposed to help with the pain of managing background jobs (and lots of other things) for Rails applications.

Here are more: https://en.wikipedia.org/wiki/Message_queue#See_also

http://web2.uwindsor.ca/math/hlynka/qsoft.html

Man, wish I had time to help you complete it, I have crawled the web thoroughly harvesting all worthy message queues, but my bookmarks are so unorganized because I used to have all important stuff in Chrome, which really has ZERO support in organizing things. Once you bookmark something, it's gone, except it's in your bookmark bar.

Then I switched to Firefox, which has better support for organizing things, starting with Tab-Groups, Tags for Bookmarks etc. unfortunately it's not helping to organize bookmarks too. Importing all chrome bookmarks leaves me with an unmanageable task of sorting things.

I had the same problem and that's why I've created this page. If you'll find something useful in your bookmarks, I'm open for any contribution, just send PR or create issue. Thanks!

What about HornetQ[1]? Qpid[2]? OpenMQ[3]? OpenJMS[4]? JORAM[5]?

[1]: http://www.jboss.org/hornetq

[2]: http://qpid.apache.org/

[3]: http://mq.java.net/

[4]: http://openjms.sourceforge.net

[5]: http://joram.ow2.org/

I like the idea of having a page listing all libraries regarding a certain topic. What's the advantage of a github + Rails solution has over a wiki page?

I'm not too familiar with Rails, but it looks like this is actually a static site generated with Ruby and hosted on s3.

I'd put redis up there too, with their blocking pop and pub/sub it's really usable as a queue.

You should add RQ in there! http://www.slideshare.net/sylvinus/why-and-how-pricing-assis...

More useful than a comprehensive list of all queuing libraries would be a comparison of the best ones with tl;dr summaries of how each works, pros & cons of each approach, and some consistent benchmarks.

I've always liked beanstalkd, unfortunately it still has no authentication support so you're unlikely to see it catch on in the PaaS environments any time soon (unless someone hooks SASL into it).

For this to be useful it really needs to be groupable by programming language.

And why it needs it's own domain/website instead of just a wiki page somewhere I do not understand.

Looks like this is getting updated, so... No ActiveMQ?

Also, MSMQ while I'm at it.

IBM WebSphere MQ

Yeah, no ActiveMQ and ActiveMQ Apollo... And the page says "Job queues, message queues and other queues. All of them.".

I think OP's motivation may involve http://xkcd.com/853.

no http://www.celeryproject.org/?

It was never on my radar. I'll search around for some resources about it. Thanks.

Site is open sourced, feel free to send PR with missing things - https://github.com/strzalek/queues.io

Celery is more of a distributed processing framework than a queue in and of itself. It usually plugs into a queueing system, but you can have it use mongo or redis for it's backend if you want and no one would call those queueing systems.

I hate to be the one that says this, but MSMQ is missing.

Doesnt specify which language lib is coded in.

Reply on news.ycombinator.com