The Big Little Guide to Message Queues
sudhir.io
sudhir.io
There's essential decoupling and accidental decoupling; decoupling you want, and decoupling which mostly just obscures your business logic.
Resilience in the face of failure, where multiple systems are communicating, or there's a lot of long-running work which you want to continue as seamlessly as possible, is the biggest essential decoupling. You externalize transitions in the state machine (edges in the state graph) of the logic as serialized messages, so you can blow away services and bring them back and the global state machine can continue.
Scaling from a single consumer to multiple consumers, from multiple CPUs to multiple machines, is mostly essential decoupling. Making decisions about how to parallelize subgraphs of your state machine, removing scaling bottlenecks, is an orthogonal problem to the correctness of the state machine on its own, and representing it as a state machine with messages in queues for edges helps with that orthogonality. You can scale things up without external persistent queues but you'll end up with queues somewhere, even if it's just worker queues for threads.
Accidental decoupling is where you have a complex state machine encapsulating a business procedure with multiple steps, and it's coordinated as messages between and actions in multiple services. The business logic might say something like: take order from user; send email notification; complete billing steps; remove stock from inventory system; schedule delivery; dispatch stock; etc.
All this logic needs to complete, in sequence, but without higher order workflow systems which encode the state machine, a series of messages and producers and consumers is like so much assembly code hiding the logic. It's easy to end up with the equivalent of COMEFROM code in a message system.
When you do use a queue as a building block in a workflow you get some technical and operational advantages, maybe those are oversold.
Will think about how to advocate business decoupling using a workflow as a the zeroth step before getting into queues.
There are two types of people in the world: those who divide the world into two kinds of people and those that don't. As one of the former, I like to view MQs as being essentially one of two types:
1. Stateless
2. Stateful
Stateless means if there's no one there to handle the message, it is simply lost. The benefit of course is you don't need to consider all the complexity of persisting messages. But a stateless MQ is really just a routing / service discovery architecture. Now that's completely fine to abstract that away but it's not really that exciting. It's still synchronous delivery, essentially.
Stateful OTOH means you really can queue things up and deal with them later. Unfortunately you're now dealing with:
- Reliability: what if the persist call fails? Do you have enough storage? Can your persistence medium keep up with the volume of reads and writes?
- Is delivery guaranteed or is it simply best-effort? This is a big one. Best effort is MUCH less problematic but doesn't tend to be what people want. You see guaranteed delivery in a lot of enterprise MQ software (eg TIBCO certified messaging) where you have things like MQ integration with two-phase commits and the like.
3. What ordering guarantees are there? Guaranteed ordering or best-effort or no promise?
Basically you've probably just created another database. Worse, that database may be different to your other databases.
Also, is your MQ a FIFO or some form of priority queue? If it's a FIFO, how do you handle throttling and buffering?
The post describes the apocryphal MQ as having infinite capacity. No MQ is.
IME the supposed benefits have been outweighed by all the negatives.
I think that's a fair assessment of persistent queues, using "database" in the most general sense.
> Worse, that database may be different to your other databases.
Different isn't inherently worse. It's just a different tool for a different job than a traditional RDBMS. The problem only occurs when we pretend they're the same.
> We can put as many messages as we want into the tube (let's assume we have a infinitely long tube) at whatever speed is comfortable to us.
and
> The receiver will never be impacted by our actions—they will pull out as many messages as they want at whatever rate is comfortable to them.
and
> Neither the sender nor the receiver are concerned with how the other works.
First, I would argue that the sender and receiver still absolutely have to know how the other works, because your API contract is now the structure of all messages that enter the queue. You also need to be aware of the semantics of both client and receiver: if I receive a message I can't process, can I get rid of it? Do I need to retry it forever? And as a sender, will my receiver retry this message or do I need to track success? In synchronous systems you have the opportunity provide clearer feedback in these scenarios, which help mitigate or at least share responsibility in mitigating poison pill scenarios.
It's also the case that when you introduce a queue you're adding an arbitrary buffer. During short transient outages this is usually ok, as your system has some slack somewhere at some time to drive down the pileup, but during persistent outages you can end up driving up a difficult to overcome buffer. And worse, in the event of time sensitive message a substantial amount of that buffer may be otherwise useless. Unless your receiver knows whether a message can be safely ignored, you can encounter outage scenarios that take hours or even days to recover from and exacerbate impact after recovery compared to if you just had no queue and shed load in the first place.
Re. size the main examples I'm looking at are SQS, Google's Pub/Sub etc which are pretty hard to fill up or overload.
Will think about how to talk about recovery measures. The handling is very application specific, it seems that guidelines will boil down to
1) if your messages are important, suck it up and add more subscriber processing capacity or
2) if they're not important, purge the queue and restart your subscribers.
Not sure I want to get into this kind level of advice without looking at the nature of the messages in the system.
It's not the size of the queue supported by the queue service that's the problem, it is the size of the queue in relation to the receivers capacity to process it. The infinite buffer can become a liability when your receivers are unable to handle it.
> 1) if your messages are important, suck it up and add more subscriber processing capacity or > > 2) if they're not important, purge the queue and restart your subscribers.
But what if things aren't so binary? Say I have a queue of emails that have backed up because my email service (or one of its dependencies) had a multi hour outage on black friday. Some of these messages are time sensitive ("We're packing it!" for a delivery that already happened) and some must be sent ("Your account is being terminated, this is the last notification"). You would like to discard messages like the former (which are certainly the overwhelming bulk of your messages) while making sure that you absolutely send the latter. Oh, and it turns out your email service has a fixed rate limit on your account and they won't raise it. This stuff happens.
Another example I've seen is a queue used as a workflow system: I enqueue commands (add X to Y, ensure X is in state Z, remove W from Y) in a backing system because, right, sometimes the dependencies for doing these operations are down for a little bit. But then one day they are down for a long-time, and the automation that has been set up on top of the external API has enqueued A LOT of commands. And a lot of those commands are now useless, things like an unprocessed "add X to Y" when there's a "remove X from Y" just a couple thousand messages later. My dependency is working to come back up to full capacity and isn't going to give me a higher rate limit, so I need to figure out how to drain this queue faster to get back to my normal line rate.
Another way to put this is that queues introduce bimodal state. When the queue is empty or only experiences temporary periods of overfill, my service runs in one way and has predictable behavior. But when my queue becomes chronically overfilled, my service is in an entirely different state. This is certainly bad news for my system, but can also be bad news for adjacent parts of the greater system that expect certain behaviors out of my service.
These areas can teach us a lot about the dangers of queues and how to handle them carefully.
Message queues are an organizational band-aid for lack of architecture and agreeable contracts between systems. RPC is always going to be a more reliable approach.
If something goes wrong with a complex system, I would much rather an exception be raised immediately and crater the caller than have a message wind up not being answered silently and having to remember to check other areas of the system after the fact.
I have worked in environments that used messaging on a massive scale, and can vividly recall having to write adhoc software to reprocess 3+ gigabytes worth of messages that failed to be picked up by another system.
A simple example would be a "Constraint Warning" message that might be produced by some factory tool. Perhaps you want to make sure all lots that tripped that warning are handled in an extra QC phase to double check things. Perhaps this is another system listening for these messages. What happens if that system fails to see these for whatever reason and you dont have a full-time employee monitoring dead letter queues? You might end up with weeks worth of broken products in your logistics chain and millions of dollars in damages before you figure out what went wrong.
Any system where you "would much rather an exception be raised immediately and crater the caller" is by definition an OLTP type system, which is a really bad fit for queues.
Lots of bad experiences with queues don't necessarily mean that queues aren't useful tools. They're just tools—they have a time and a place. If you've tried integrating with an email or SMS gateway, or a legacy rate limited bank you might have a different experience. Or dealing with slow job execution combined with spiky high speed job creation.
Depends entirely on the problem, and probably the solution team and constraints as well. Don't think it makes sense to dismiss them out of hand for all problems, though—just as it doesn't make sense to claim that they should be used for each and every problem.
For instance - You have 2 services: Bank Core Integration and Teller. The Bank Core Integration service contains the queue of items that need to ultimately be pushed to the underlying legacy, rate-limited system. The teller service operates in RPC terms with the bank core integration service, so callers into the problem area are isolated from the semantics.
This might sound like a subtle difference, but it is huge in practice. If there is some problem with that queue, it is entirely contained to that one service and can be dealt with in isolation. Logs from that one service should comprehensively document the concern. If there was a message bus between these systems, you would have to go back and review message logs to see if things got missed between systems.
Implementing both the message queue and the bank core integration service within 1 service makes for less communication/network overhead and is easier to debug, which is definitely valuable.
Seperating them out means you can use off the shelf products for the message queue and is easier to scale, which can also be valuable.
> What happens if that system fails to see these for whatever reason and you dont have a full-time employee monitoring dead letter queues?
"What if you fail to monitor correctly" is a bad scenario no matter the architecture, but doesn't critique message queues. Any sensible message queue will have lag and backlog metrics on subscriptions, which you can monitor just as easily as an RPC end point.
Seems like an odd statement. RPC seems considerably less reliable to me since it ties your business logic services states' together.
> I would much rather an exception be raised immediately and crater the caller
Why? The caller might not care. And what if you have a large graph of callers? You're going to end up with a cascade of failures, when really none of them are going to do anything except retry or drop - something that the failing service is capable of making the call to do in most cases.
> I have worked in environments that used messaging on a massive scale,
Same, and in my experience it was the RPCs that caused the most trouble. I don't think anecdotes are going to be helpful here.
> Perhaps you want to make sure all lots that tripped that warning are handled
If you want to make sure of something, and then you don't make sure of it, I don't see how that's the fault of a queue. Just... check?
I spotted one small typo and thought you would like to know:
“like transferring information form your software into an email or an SMS on the cellphone network.”
One thing you might consider adding are more examples for different popular queuing systems and how they differ from one another. The software I always reach for is nsq (https://nsq.io/) because it's meant to run co-located with the message producers and readers are supposed to connect to multiple instances where the messages are produced (using a lookup daemon). This is quite different from the queues on your list, so much so that I'd consider adding it just because it works so differently.
Talking to a bunch of engineering teams I found that some use case for queues are very generic (almost identical use case and implementation across teams). Specifically webhook handling is something that keeps coming up. We've been working for a few months of a queue that's specifically for ingesting and delivery of webhooks. Do you see a future for use case specific queueing systems instead of defaulting to a general purpose queue?
In our case we abstract the actual implementation and behave more like you would expect a standard webhook.
For reference, it's https://hookdeck.io
This was possible only because AWS provides these services, of course. If you’re offering an infinitely scalable HTTP endpoint to soak up webhooks and allow me to query them at my leisure, or put them into a queue for me, that would be useful.
I haven’t looked into hookdeck in detail yet, will post again once I do.
We've heard from teams having issues dealing with large uncontrollable spikes from their webhook providers and we can smooth out that out. There's additional benefits that can be introduce before it gets to your own infra such as verifying signatures, filtering events, etc.
API Gateway + SQS + Lamda is definitely a common and good approach. My understanding is that you often start running into into other problems. Hitting DB connection limits from serverless invocation is a recurring one! I'm hoping we can make the troubleshooting / replayability easier as well.
Thanks for sharing your approach and opinion! Hoping to hear more!
Anything specific you think is worth pointing out?
That pattern is really powerful and simple, even if the main tool using it (Kafka) ends up being difficult to operate.
https://www.brightball.com/articles/organizing-background-wo...
Amazing how people sucked down this rabbit hole. My work is doing this right now, and its driving me mad as there are well supported alternatives out there. But I suppose there is something intrinsically fun about building a complex communications system.
When I asked for the rationale, they said that x queue system didn't do ordered delivery. The queue system literally has the word "ordered" in it.
Its not really their fault, they project manager should have stopped it. However its still not an excuse to re-invent the wheel.
This article is inconsiderate of the added complexity of MQs.
How is the payment processor given as example decoupled from the main process from let's say an e-commerce website using a MQ ?
If the payment processor isn't responding, the payment process on the main site is broken and a client can't purchase.
If many clients try again to pay, the MQ will be flooded and attain its max limit. How do you deal with added complexity and extra error handling of the MQ ?
Same question for integrating a new payment processor: with a different API or SDK to integrate, how would that not be the case with a MQ as stated in the article?
A MQ is simply a buffer when you need to scale up service consumption. But services with APIs don't need it, they do rate limiting and if they're down you should better set the status on your program and stop communicating until it's up again.
MQs are not automatic and this article is selling false promises and examples.
Im not sure what you are talking about here. The two concepts are quite different and are meant to be used for different processes.
> selling false promises
AFAIK, the article wasnt selling anything. I think it was a nice intro and overview to message queues.
MQ's are distinct from APIs. APIs require both sides to be free to talk, while an MQ allows one side to send when it's able and the other to read when it's able which can happen out of sync.
MQs do have limits but this is typically several orders of magnitude higher than an API - a few million queued messages is pretty trivial on any decent MQ cluster.
An MQ isn't a buffer alone, it's also routing, retries, error handling, deadlettering, etc. This is the difference between your crappy in-memory blind buffer and something like RabbitMQ. Yes MQs still require maintenance/attention/etc, they're not magic.
I don't think this is quite right. An API might use a message queue as a transport mechanism.
The MQ only gives you a way to communicate, it doesn't define how you structure your requests.
in your payment example, if you replace MQ with REST, it gives you the same answer.
Some message queues give you rich diagnostics for when messages fail. Others like NATS only give you the guarantee that the server will be up.
MQs come in many flavours. some act like python's Queue, but allow you to use many machines.
Some are as you say, a buffer.
However I like to use them for connecting n transient instances of a service to an API front end. The routing is handled for you, and you can isolate services from each other.
So perhaps I'm bitter but, while they seem useful for specific purposes, it does feel a little religious and one of these presumed 'must have if we're serious' class of program when other solutions which architecturally resemble them might suit the problem far better.
In my current job we interact with kafka and have definitely encountered many less than ideal aspects of its behaviour. But I guess that must be the junior developer talking again ;)
Back in the days when we didn't know what we were doing, we had a system where MQ was used in Request/Response style for querying databases. On the beginning of this was a web endpoint. Frequently we end up in a cascading failure whenever we hit an overload situation, because the web client would time out, and retry another call. Meanwhile, there's a bunch of query messages enqueued to be executed against the DB despite the fact the client had already disconnected. If you are trying to web scale, you'd end up paying for IO twice, first to query the DB and second to persist the results in a queue. The big take away for me was to be wary of coupling an unreliable synchronous RPC with a reliable asynchronous MQ.
Back-pressure would be a useful concept to add.
Back-pressure here would mean that emitters of messages into your queue behave differently when there is an excessive number of messages waiting to be processed.
For example:
- If it's a job queue, load-shed less important jobs until the queue is healthy.
- If you are ingesting a high volume of messages, be more aggressive with batching them together before inserting as a single queue message.
Think of the queue-as-message-bus pattern -- very common. You don't want to have a service with millions of pending messages to process, it forces you into painful, slow recovery, or making the tough choice of dropping everything.
Anyway, great write up, and happy new year!
> This is the holy grail of messaging, and also the fountain of a lot of snake-oil.
Exactly once message delivery is quite possible with messaging systems that support transactions. When combined with other transactional resources (e.g. database) and a distributed transaction monitor, exactly once messaging works well and is rock solid reliable. The grand-daddy of message brokers IBM MQ is absolutely capable of exactly once messaging.
There’s a note in the next paragraph about how systems manage to say that if you pass in the same message ID / token for X minutes they won’t be duplicated, and my ensuring FIFO there’s a side effect of not giving out the next message until the current one is acknowledged.
This leads to a situation where there’s a guarantee of exactly once acknowledgement, but not necessarily exactly-once processing or delivery. Given that the semantics of at-most-once and at-least-once apply to processing and delivery, I personally don’t think the goalposts should move on exactly once.
Systems claiming exactly-once lull developers into not planning for multiple deliveries on the subscriber, or the need to do multiple publishes, both of which can still happen.
When people say “exactly once” is impossible they really mean in the face of failure at the queue level.
And what exactly is impossible with that? Just wait it out, i.e. like all the CP systems do (as per CAP).
Note none of this is rigorously defined either in the article or with most message queues and the configuration of queues/brokers/clients means that there are all manner of edge cases around delivery guarantees in practice.
As an example, imagine a system sending a million 1 KB messages per second. To survive a 1 minute network outage it would need 60 GB extra storage to park the messages. If the outage lasts longer than it has space available, dropping messages becomes inevitable.
In practice I agree that these problems are quite rare since most network are reasonably stable. However, especially at scale it's not rare to see messages dropped or delivered more than once. I have no doubt IBM MQ can achieve exactly-once most of the time, but no distributed system can achieve exactly-once delivery all of the time.
That is not correct. All interactions between the client and the broker are performed in transactional units. If the transaction in which messages are sent fails to commit, then the messages are not sent, and all work is rolled back. Once a message is successfully send (that is, sent and transaction committed), it will be delivered once and only once to the receiver.
Likewise on the receiving side, a message is delivered and the encompassing transaction is committed once and only once. A message may be delivered more than once if the encompassing transaction is later rolled back due to say network failure. But a message delivery in a transaction that does not commit is not a delivery.
The benefit here is that application programmers don't need to concern themselves with message duplicate checking and the risk that duplicate checking is done incorrectly leading to bugs that are very difficult to identify.
But even if it does join, that's still not sufficient, because these systems can become unavailable during partitions, and that is definitionally incompatible with "exactly once".
Richard Stevens book APUE, seems to not like Message Queues, as much as FIFOs, especially IIRC, FIFOs work on file descriptors and therefore works nicely with poll/select and also since destruction of FIFOs is handled more gracefully as the connected programs terminates. Is that view still holds good?
I wonder if that view still has currency?
Of course .NET can be replaced by anything that supports async, non-blocking functions.
If instead I used message queuing instead of writing the code myself I'm adding yet another configuration item to my system, replete with security holes, patches, operational management, and sourcing the skills to procure, install and run it. Of course each has idiosyncracies you need to learn and understand - like Rabbit's memory-based queues. When the instance goes, so potentially does reliable messaging.
I'm very much not a fan of message queues. Haven't been for a good many years.
EventBridge isn't a queue either, actually.
Not necessarily. Nothing stops one from using Kafka topic in a fashion similar to a queue but even then, the way this would have been done differs significantly.
Very simplified: in a queue, an available message is picked up by a consumer and must be acked / nacked within a certain period of time. If message is acked, it is considered processed and may be chosen to be removed, depending on the queue semantics. If nacked, it becomes available for reprocessing. Multiple consumers may be processing messages at completely different offsets and a failure of processing at an early offset does not prevent other consumers advancing the queue.
In a commit log, if the consumer is not able to process the message, it has to keep retrying that one message until it can be processed OR ignore failure and publish the message back to the log at a new offset for future reprocessing. Regardless of that, any other consumer (or group) has its own view of the world so two consumers (or consumer groups) will process all messages from the topic.
The semantics are completely different.
It's possible to guarantee exactly-once delivery, just not in bounded time, but eventually. That's what FLP impossibility is about, impossibility to achieve consensus (e.g. exactly-once) in bounded time. But if you are talking in the context of semantics, it's a totally different thing, it's a computation model and it's absolutely possible to have exactly-once semantics, regardless whether you achieve it by waiting, like consensus-based systems, or by using special data types and operations that can be performed without waiting and eventually converge, like CRDT systems.
I hope you'll find them as useful as I have in shaping my interaction with this community. Specifically I'd like to highlight the theme of substantiative comments and the discouragement of commenting on small issues like website formatting.
I understand that it's a bit on the nose to see user account sudhirj submitting a link to sudhirj.io. However, disclosures are also a clear indication on where to post questions and/or remarks directed at the author.
Anyway, my 0.02€.