Work queues when using messaging

From my experience, if you're fine with pushing a few dozen tasks into a queue, then you're likely fine pushing a few tens of thousands into the queue as well. You won't be able to go faster than the max throughput of whatever lies downstream of the queue, but you won't break anything either.

If the tasks in the queue are not important, then it's fine if they are dropped.

If the tasks in the queue are really important (to the point where dropping one will require remediation) then you would already have set up the necessary tools for introspection and control, even for a few dozen tasks. In particular:

- Detecting that a task has been lost, so it can be re-inserted later.

- Detecting that a task has stalled, so it can be removed and investigated.

- Reordering the queue based on the relative priority of the tasks.

In any case, I would argue that if an upstream system needs the ability to hold some tasks back when the queue is too full, then the queue is in a bad middle ground where it's not completely solving the problem (since "holding back tasks" means a separate queue-like feature in the upstream system) but still consuming a chunk of the infrastructure simplicity budget.

From there, one should either eliminate the queue (replacing it with the "holding back tasks" feature in the upstream system, combined with backpressure from the downstream system) or improve it until it can accept everything the upstream system throws at it.

I feel like you are talking about ‘MQ’ in general term, but you’re describing a pretty specific one. One that is not very flexible and requires you to bend your system around it to make it work.

Not all MQs are the same. Some offer a plethora of options to fit your need (ack policy, replay policy, max outstanding, etc) Give NATS a look: https://nats.io/

I'm not sure I agree with the article. It seems to argue against the point of having a message queue and suggests doing a bespoke reimplementation of a queue using a database.

However, all of the things listed in the explicit work queue section can be done using regular message queues as well, so...why bother reimplementing it yourself?

Author here. Some comments to the comments:

Yes, I do effectively argue for (re-)implementing a queue system (for certain situations), even though you already employ a message-oriented architecture! But this is not necessarily meant as an explicit, separate DB table: If you have an orders table, you can use that table as your "Work queue". So, if some event makes 10k orders fulfillable, you will now have to fulfill these, and if you use a queue system, that might entail putting one message on the queue for each order. You could now choose to stick all those 10k jobs on a queue in one go, or you could pace them out. I argue that I would go for the explicit pacing if the fulfill process is complex, consisting of multiple hops between systems, and hence queues - and I try to put forward some arguments in the article.

And yes, the "fabric" of messaging endpoints should really be able to handle any number of messages on the queue. The downstream processors should already have their concurrency (number of parallel consumers/executors) tuned so that they do not trash the external resources like databases. However, several of those sub processes, or the external resources they touch (e.g .databases), are probably used by other processes during the day. Some of these might be human interactive like looking up a customer. By throwing out all the orders on the fabric in one go, you might still make the system annoyingly sluggish due to interaction between all the concurrent jobs: There will probably be a difference between "flat out" processing of hundreds of thousands of jobs, and what happens otherwise throughout the day. Having an ability to control this from the top is nice. And I do point out ways to achieve such pacing using just the MQ (e.g. use a choke-point to pace), but I would still prefer explicit pacing, i.e. explicitly put smaller subsets of the total 10k orders onto the queue, since this also gives other nice effects:

The most important points are the ones about introspection, monitor and control. These are elements which I feel are essential to feeling confident about the system health, and in particular for being confident when you for example make larger refactorings to the process. Yes, you can have monitoring and introspection by having the order table be updated by the processing of each order (i.e. update some field, describing which stage of the processing it is in, and if you get error conditions). But by just throwing all the orders on the messaging fabric in one go, your reduce your ability to explicitly halt or abort if some day things start failing, or to easily tune down the pacing after a large refactoring just to be extra careful the first time it runs in prod. You can even throw in an automatic halting: If e.g. more than 100, or 2% of current orders, fail processing, then stop the issuer. Yes, you can use your logs to see what went wrong, but it is nicer to have just a few such processes to understand, than 10k processes that failed, with redelivery-retries kicking in, and overall general massive mess.

Wrt. the argument that the messaging system could give you this control: None of the classic MQ systems, like ActiveMQ, Artemis or RabbitMQ, gives you good domain-specific introspection abilities, that is, see the orders, customer info etc. And that is how I like my MQ: I want it to be a extremely efficient and robust, but rather dumb, message broker. I want it to pass messages. I do not want the message broker to orchestrate and control my processes - that I want to do myself. Mats3 (https://mats3.io/) is an attempt to make such orchestrating of multiple steps simpler when using messaging - as simple as coding it in a sequential, top-to-bottom coding style, while still having each step be fully independent and asynchronous, reaping the benefits of a message-oriented architecture.