Show HN: Martinet – Distributed Task Queue for Node.js | Hacker News Reader

Show HN: Martinet – Distributed Task Queue for Node.js

github.com

28 pointsmpconlen9 comments

Generally speaking, using a database as a queue is a particularly bad anti-pattern.

Using a database as the backend of the queue is a really good idea. Of course if you have a lot of load you can add a redis cache in front, but persistence and reliability of the queue data is highly important.

If not a DB, how else do you get consistency, reliability and ease of development? Basically you will end up re-implementing a database anyway.

PS. I wrote a highly popular render manager / queuer and I first ended up writing a custom database, but eventually the core was replaced with MongoDB and it is now even more scalable and robust.

I guess this should be clarified more in the README - this uses zmq (and potentially other brokers) as the queue, and uses the database for tracking results, saving recurring tasks, etc

Good to know, so more like a crontab entry for the database.

Where's the actual queue though? zmq isn't a message queue, it's more akin to a protocol. e.g. if I have N workers and M jobs, and M>N, where are all the >N indexed jobs kept while the others are being processed?

Like it appears you're using PUSH/PULL pattern, which means the jobs queue up in the publisher's memory (on a per PULL-peer basis..). So if the publisher crashes, those jobs are lost. If the number of jobs queued exceeds the high-water mark, those additional pushes are either going to discard messages or block. If it blocks, it looks like your code is going to have lots of other issues... If the worker crashes, I believe that queue is also going to be lost.

You are right about the push/pull pattern, in the case the publisher crashes those jobs will be lost. That could be an issue depending on the use case and would be a great thing to address.

If I am understanding the other notes correctly:

Neither the worker nor publisher should ever really block (even in cases where M>N, N>M, publisher crashed, etc), mostly due to the async zmq bindings for node. If there are a ton of messages being sent to a single worker this worker will pull down messages as fast as it can and the rest will still be in publisher queue.

If a worker crashes this is happening separate from the publisher and won't affect the queue stored in the publisher's memory

Why?

Redis is a "database" and is used as a queue for many different things. kue.js uses redis for its job queue. Agenda.js uses mongo for its job queue (which has a very similar API to this one listed).

The context was a RDBMS, not Redis. Redis makes a perfectly suitable queue, I agree.

The problem with using a RDBMS as a queue is it results in a continuous polling loop, it's incredibly slow, guaranteeing only a single subscribers gets a message is incredibly complex (you need to employ locking mechanisms yourself), etc.

Using Redis you can just BRPOP and it's perfect.

How is this different from beanstalkd ? From the README file persistence seems like its main feature, but beanstalkd already has this (via a write-ahead log), along with priority, delayed jobs, burying/kicking jobs, TTR, explicit ACK (for distributed workers), etc.

Just use the nodejs api for beanstalks.

Reply on news.ycombinator.com