Hacker News Reader

Comment by danielstocks | Hacker News Reader

I used RabbitMQ together with python and celery quite extensively and it scales really well. One thing we had trouble with though was to find a nice mechanism to scheduled tasks. Eg. “Run this task 12 hours before departure”. Maybe AMQP is the wrong place to solve that problem.

View on HN

I've been using something like this for exponential backoffs, but I think it'd work for this case as well.

Let's say you've got one exchange and one main queue for processing: jobs.exchange and jobs.queue respectively.

If you need to schedule something for later, you'd assert a new queue with a TTL for the target amount of time (scheduled-jobs-<time>.queue). Also set an expiry of some amount of time, so it'd get cleaned up if nothing had been scheduled for that particular time in a while. Finally, have its dead-letter-exchange set to jobs.exchange.

This could lead to a bunch of temporary queues, but the expiration should clean them up when they haven't been used for a bit.

Haven’t used this myself, but there seems to be a plugin for delaying message delivery: https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages...

You schedule tasks very easily with the Celery Beat Scheduler. I've used it in production system to kick off big jobs and notifications.

https://docs.celeryproject.org/en/v2.3.3/userguide/periodic-...

You're usually stuck polling for stuff like that. I'm a big fan of using things like Advanced Python Scheduler for those sorts of tasks: https://apscheduler.readthedocs.io/en/stable/

Yeh I would argue don't use a message queue for this, they're really best processing many messages quickly, there are plenty of scheduling libraries that have various persistence layers to handle this depending on your ecosystem.

Celery has eta/countdown params that allow for running tasks at a specific time

you can use a redis sortedset https://redis.io/commands/zadd

Reply on news.ycombinator.com