Hacker News Reader

Comment by jalfresi | Hacker News Reader

This appears to be similar to beanstalkd but with some nice additions (scheduled time for job to run, jab max existence time).

One of the things I awlays wanted beanstalkd to have was an atomic move-tube command, so you could emulate a state machine using queues and tubes

View on HN

That is correct, beanstalkd was an inspiration for this project. I've even credited it here: https://github.com/iamduo/workq#credits. Could you describe what the use case of the move-tube command was? More specifically on what it was trying to accomplish within the state machine?

Internally Workq, for simplicity, does not have any separates "tubes" however. Just a job pinned by its name.

The idea was that you would reserve a job in a tube, do some work, then move that job to another tube upon success atomically, without the need to delete the job, then create the job in the new target tube. The problem is that if you create the job in the new tube before deleting, the TTL could kick in and return the job to the original tube, meaning you have the same job in both tubes. The other alternative is to delete the job then put it in the target tube. However, if your worker process dies during this step, you may end up losing the job.

It sounds like what you need is some sort of transaction support so that deletes and adds only happen if both succeed.

Which would make things a lot more complicated - vs. issuing a move command on a job id where the server would be responsible for making sure the atomic move succeeded.

It's not a difficult feature to implemented (in fact if I recall there is a pull request open for this feature in beanstalkd) and IMHO would open up a lot of interesting use cases.

Is there a reason why there aren't 2 types of jobs? One for each stage? You mentioned the first stage there is some work performed, then it sounds like you need to pass on the work again to another worker for the second stage.

Reply on news.ycombinator.com