Hacker News Reader

Comment by _verandaguy | Hacker News Reader

I used this in a project at a previous job -- I have to say, while the API is simple and useful enough for small projects, it raises some issues with how it's designed.

Instead of relying on trusted exposed endpoints and just invoking them by URL, it does a bytecode dump of task functions and stores those in Redis before restoring them from bytecode at execution time.

This has a few drawbacks:

- Payloads in the queue are potentially a fair bit larger for complex jobs - Serialization for stuff that has decorators (and especially stateful one, like `lru_cache`) is not really possible, even with `dill` instead of `pickle` - It's not trivial, but this exposes a different set of security risks compared to the alternative

I don't want to say this is a bad piece of software, it's super easy to set up and way more lightweight than Celery for example, but it's not my tool of choice having worked with the alternatives.

View on HN

I had to setup a task queue in an old, resource constrained environment once. I ended up forking the client process into a worker when the queue wasn't already running. Any time we need an expensive method call to be throttled asynchronously we just have to add a decorator to it.

https://github.com/deckar01/low_queue

Wait is that how RQ works?! How does it handle code updates? What happens if I update the task code while there are tasks still enqueued?

Can't speak to the code updates since I never went that in-depth into the code.

The pickle logic lives here: https://github.com/rq/rq/blob/80c82f731f57186f8c97239b38100f... so if you've got a minute you can skim that and hopefully get an answer to your question.

Which alternatives worked for you?

Can't speak for op, but for us the ones based on RabbitMQ have performed well:

https://github.com/Bogdanp/dramatiq

Stay away from Celery if you can, or stick to v3.2 because 4.x has a ton of bugs.

Celery has always been a massive resource drain, and for pretty much zero gain. I'd suggest just saying "stay away from Celery" as it's easy to get going with, but really hard to scale or work with when you need what it promises. There are better options.

Agree that you should stay away from Celery in most cases, but it does integrate well with Django, and pre-4.x is stable enough.

It's just a fairly higher amount of boilerplate and setup compared to most of the alternatives either way.

So far I've mostly used Celery in Python, but at jobs past there was also Sidekiq (granted, the use cases for that were a bit different, but it's kind of in the same bucket).

Haven't had the opportunity to work with them, but I've heard good things about SQS and other cloud queues, but I think those are fundamentally message queues, so you'd be responsible for writing a job queue layer over it, which can be a huge hassle.

Reply on news.ycombinator.com