Ask HN: Why are there so many queue services but almost none support scheduling? | Hacker News Reader

Ask HN: Why are there so many queue services but almost none support scheduling?

16 pointsjjeaff17 comments

I have recently gone down a rabbit hole trying to find the right queueing service for dealing with scheduled tasks. I'm a bit befuddled as to why there are so many queue-like services and applications out there that all seem to have near feature parity.

There's AWS SQS, Google PubSub, Azure Service Bus, Kafka, Nats, IronMQ, StormMQ, RabbitMQ, ActiveMQ, ZeroMQ, even Redis and so many more.

And yet, not a single one of them seems to support scheduling as a first class feature. Some (like Google pubsub) mention hacks like de-queing, checking a stored timestamp then re-enqueing if the time hasn't passed yet. But that just seems crazy. (I guess it makes a lot of sense if you are charging by data transferred). SQS allows you to delay a message for a maximum of 15 minutes.

What solutions are out there for robust scheduled queues? There must be something.

View on HN

I'm the author of Sidekiq, which supports first-class scheduled jobs. It's designed for Ruby apps.

https://sidekiq.org

I'm also the author of Faktory, which supports first-class scheduled jobs and is designed for applications in any language.

https://github.com/contribsys/faktory

Both are open source and both have commercial support so they will be maintained for years to come.

Thanks. I will take a look.

If you only need to schedule for a handful of different delays, you can do so with RabbitMQ by having a queue with no consumers and a certain timeout with a “dead letter queue” configured. Messages will stay in there until they reach the timeout and will then be delivered in the dead letter queue. I’ve seen this approach being used successfully before.

Yes, but I think a common use case would be scheduling reminders, notifications, and emails for users. Which runs into the millions of scheduled tasks at different intervals pretty quickly.

For a handful of fixed intervals however (like notifications), the RabbitMQ solution works well - have one queue for "In an hour", another one for "in a day" and so on.

For completely arbitrary intervals you could look at Celery and its "Celery Beat" scheduler, otherwise for a proprietary solution look into Google Cloud Tasks which seems to have a "scheduled_time" parameter (haven't tried it personally though): https://cloud.google.com/tasks/docs/reference/rpc/google.clo...

Use the SKIP LOCKED function in your SQL database.

You then get a huge amount of flexibility and it becomes easy to implement any custom queueing mechanism that suits you.

To expand on andrewstuart's comment:

Consider PostgreSQL. PostgreSQL' skip locked functionality lets you roll your own queue using the database. There's a good article here: https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock...

I used this as the starting point to build my own custom queue-thing using postgres & skip locked with the following features: there is a fixed collection of about 100,000 tasks that need to be executed roughly on schedule, each task can have a different schedule and a priority, tasks can require certain resources to be available as a pre-req before they can execute, and finally there's a mechanism for tasks to define dependencies on other tasks. If a dependency of a given task executes, then that task is also scheduled for execution.

Maybe "queue" is less of an accurate description than "incomplete, bug ridden, ad-hoc, poorly specified implementation of make + cron". But there is task-queue like functionality so worker processes can do queries like "lock and give me the highest priority task that is scheduled for execution or overdue and that has enough resources to execute".

I don't have very many tasks (only 100k) and the update frequency is weekly or slower for most of them. I get reasonable performance running both postgres and two python worker processes (that execute the tasks) and a web app on a single low-energy box with an AMD G-T56N CPU and about 4gb of RAM. Cost is about $40 for the hardware (second hand), $15 a year in electricity, and $0 = $0 / hour * 10 ... 40 hours for the custom software build and a few post-release debugging sessions in order to get the locking working smoothly, since I don't bill myself for software dev services.

Exactly this! You can have column like "occurs_at" an then filter with "occurs_at < current_timestamp" together with ordering on that column or possibly some priority one too to have workers perform these tasks in importance order.

Retries are trivial too - you can have attempts countdown column etc.

Also check NOTIFY/LISTEN which can be used to avoid pooling and wake-up workers efficiently. https://www.postgresql.org/docs/current/sql-notify.html

There are libraries based on Redis, however since Redis 5 to have scheduling is simpler, because now we have ZPOPMAX and ZPOPMIN and the blocking variants as well. So add the item in the queue as a sorted set keyed by the unix time, and pop stuff. Using some scripts it is possible to also backup the item currently being processed in order to be resumed on failure, however this is yet not a first class feature like it is for lists (see BLPOPRPUSH).

> de-queing, checking a stored timestamp then re-enqueing if the time hasn't passed yet

That sounds reasonable to me. Any condition that means you can't process that yet, just re-queue it for later. Let the Queue be a dumb single purpose component and do what it does best.

If you are upset about making the business logic more complicated: The thing that does this check could be middleware so that your application code can not be concerned with it.

For the philosophy see: http://harmful.cat-v.org/cat-v/

> "cat isn't for printing files with line numbers, it isn't for compressing multiple blank lines, it's not for looking at non-printing ASCII characters, it's for concatenating files."

So you could argue "queue isn't for scheduling messages, it isn't for X.. Y.. Z.., it's for managing a queue of things"

Azure Service Bus supports scheduling messages for future delivery. It's a first class citizen via the API when you drop a message in a queue. You can see https://docs.microsoft.com/en-us/azure/service-bus-messaging... for more details. You can also use Azure Scheduler which integrates with Service Bus to directly drop a message in a queue at a specific time on a reoccuring schedule.

I think the queues you mention at solve a different problem. Maybe I’m missing what you are trying to do but setting up cron is the way to do scheduled tasks on Linux.

Azure storage has a visibility timeout of up to 7 days.

https://docs.microsoft.com/en-us/rest/api/storageservices/pu...

Like you I wish this functionality were first class; it seems like it will work, but it's not emphasized in docs. And the alternative of creating a separate service seems silly.

If you are using Redis and ruby, Sidekiq offers scheduling. I am unsure why a lot of startups insist on using the far inferior resque instead.

Qless supports both delayed single jobs, and recurring jobs with an interval.

At it's core Qless is a bunch of Lua that's executed in Redis by the binding you choose.

If you can get past the futzing around to have HA and secure Redis, its a nice setup.

Some of the queue consumer libraries support scheduling. Celery comes to mind.

the current state is far from optimal. Some less AWS options are:

- Use DynamoDB TTL to trigger a delete and listen the dynamodb stream.

- Use step functions and pause execution for up to a year (1 million max concurrent executions)

Reply on news.ycombinator.com