Comment by eropple | Hacker News Reader

Comment by eropple | Hacker News Reader

Parent Full thread

Whee! My first Show HN. I've only been here, like, seven years and all.

But, yeah, I've been tooling away on TaskBotJS in my metaphorical garage for a while. I've found Kue and Bull really frustrating to use (no shade intended; they feel like they made a lot more sense in a pre-ES6 universe?) and `node-resque`, while scratching a little more of that itch, just made me miss the comfortable guardrails of Ruby offline job queues. So I made one.

As my day job is devops and devops accessories, making it easy to run and administer TaskBotJS was a big priority for me and I've tried to get the best of both worlds: mostly turn-key to run in simpler cases but less prescriptive and more pluggable at scale. TaskBotJS is pretty extensible and supports job middleware and event listeners; I have a Sentry plugin coming in the near future as an example (and also 'cause I like Sentry).

I also like to write documentation[0]. While it's not yet complete, what's there is pretty exhaustive and I try to explain my thinking as well as just "what it does"; capturing the whys is as important as the whats, to me, to better foster conversation around the project and to make for more constructive issues when systemic, design-focused concerns are raised.

It's running in at least two production environments (only one of which is mine; I dogfood but I wouldn't have released it as a "you should check this out" version if somebody else wasn't checking me) and is being rolled out to at least one more in the near-ish future.

Thanks for taking a peek, and I'd love to hear any actionable feedback that comes to mind.

[0] - https://github.com/eropple/taskbotjs/wiki

Congrats on shipping! As someone who's written similar software[1] (for Python), I can appreciate the hard work that went into this.

One piece of feedback regarding the docs: the documentation section in your README has two links, one to the wiki and the other to the website. I had initially scanned the section and followed the link to the website (I don't automatically associate "wiki" with "documentation") and was then unable to find your documentation for a good couple of minutes. You should make the wiki link more prominent.

[1]: https://dramatiq.io

Nice! We need more work in this space. I'm one of the node-resque authors. I'd love to hear more about what you felt was missing from the ruby version... clearly we copied a lot :D

I'm using Kue heavily and so far had no problems using it. Can you elaborate on 'pre-ES6 universe' or any other issues you had with it?

TaskBotJS looks great. Does it support recurring scheduled tasks?

For example:

  run task foo at 13:00 every day for 14 days.
  run task bar at 06:30 every day forever (until cancelled).

I love Bull and wrote a front end for it. I've had Bull running for years with no problems and minimal management. Can you give some specific details on the problems you found with Bull? Writing a queue that atomically processes items is non-trivial, so when I wanted a feature in Bull I just improved it, the devs are very receptive.

Otherwise it seems like TaskBotJS is just NIH so that it can be LGPL and then commercially licensed for features that Bull has for free? This licensing setup means I will probably never touch TaskBotJS.

EDIT: I see there is a test script line in package.json but I don't see a CI badge and can't find the tests in the source? The first thing I did with Bull is verify it with the test suite and then add more tests to ensure consistency across various job types and errors. This is critical for a background job package which is meant to be left alone.

If I had to choose between Kue and Bull, I'd choose Bull. But I found it frustrating to work with.

- I want to support multiple backends. Bull is a Redis queue. Which is fine, as far as it goes, but I have some early designs on an AWS-backed SQS/Dynamo backend that obviates the need for Redis entirely, among other things. ElastiCache is not the worst thing in the world to have to run but not having to directly run anything is, in many environments, preferable, so that's a route I want to go down.

- Separate queues for each type of job (per-key) isn't sufficient control. I think it's better to categorize jobs by priority. And while one could multiplex on top of a single queue it, tbh, makes it pretty rough to handle stuff like "these jobs are all of similar priority but have different retry counts and strategies". That multiplexer then isn't integrated with metrics, frontends, etc. and it becomes a "I guess we just reimplement everything except the basic transport on top of Bull" situation.

- Also, the basic transport itself is arguable. TaskBotJS uses BRPOP in Redis and I think it's better in the general case to go "if the process hard crashes and doesn't have time to respond to a stop signal I may lose a job". When necessary, an administrator can explicitly opt-in to more complicated and more resource-intensive completely reliable queueing. IMO, RPOPLPUSH plus a heartbeat is a better way to do this than expiring locks, too; expecting a developer to renew the job lock or tune its timing is a bigger ask than "the worker's still working on it--whoops, the worker went away, requeue the job".

- Bull uses functions over classes for jobs. In many ways that's defensible; I don't think it scales (people-scales, not computer-scales) across a large dev team without doing a lot of work on top of it. TaskBotJS expects you to provide dependencies to your jobs that it can inject into the worker state--every Kue/Bull-based processor I've seen tends to use globals wrapped in the job closure for stuff like database connections, and I do not subscribe to this point of view and I think you shouldn't either.

Any one thing (save pluggable backends) I could have implemented on top of Bull. IMO it'd have been worse to do so, but I could have. But the sum total of things I think could be done better made it easier to write my own, and then to refine it over time into TaskBotJS.

Don't get me wrong: I think Bull is fine. The compromises it forces upon you are ones I don't want to make and I don't think developers should be forced to, but of course you can get by with it. But I wouldn't have built TaskBotJS if I didn't think it was better. Throw shade if you'd like; I made the tool before I made the product and I really don't expect it to make more money than enough to keep me focused on its maintenance.

My priorities as a business needing a job queue are:

- Does it run my jobs reliably?

- Does it run my jobs with good performance?

- Are failed jobs obvious and simple to report/retry/log?

Bull's tests and history shows that these needs are addressed.

For a backend, the Redis strategy employed by Bull is battle-tested and used by many other related projects. I can't say that about any other backend. I haven't found this to be a common, real issue in practice.

For multiple queues, I can easily run multiple namespaced Bull instances, so that is not an issue.

Bull uses subscriptions and RPOPLPUSH, so I'm not sure where the polling problem is that you mention. Bull's performance testing is very competitive.

Functions vs classes is an interesting dev choice... In JavaScript there is really not much of a difference. I would argue that functions are more lightweight, which is good for a job queue, but I don't actually think a job queue needs an opinion on client JS architecture.

If you have performance numbers and a test suite for TaskBotJS, that would go a long way to support the claims.

My priorities are similar, but #1 for me is how easy it is to maintain. It sounds like most of the benefits over Bull are focused on that area.

That's a fair read, IMO. My contention is that by the time you've built something on top of Kue or Bull that feels good to a developer...you're probably going to have built most of TaskBotJS.

I'm planning to use a similar model OSS / commercial as well. Mike and Sidekiq is my inspiration for this. Good luck going further with the project and congrats for the launch!

sorry i don't get it. why would somebody use this over a rabbitmq.

I think it is a legit question... I poked a little around and I did not find an answer.

Read and find out

Reply on news.ycombinator.com