How far can you go without a message queue?
blog.chiselstrike.com
blog.chiselstrike.com
Don't add a message queue until you really need one. And if you do make sure you account for it being down, running locally, back pressure if the queue gets full, monitoring, and logging at least.
So, you can go very far without a message queue... by using a simpler message queue?
Lets draw the line somewhere the other side of applications that advertise themselves as having that function?
it’s basically a cache with a fancy API so you can incrementally update data structure instead of having to constantly write the whole thing you are trying to cache.
If you mean “it doesn’t have every feature of Postgres” and behaves a little differently then sure but it absolutely is a database.
Watch is a poor man transaction and stored procedure does not work when you need to read value interactively from somewhere else (user/filesystem/another database) while inside a transaction.
I don't disagree with you about WATCH being less powerful than PG transactions. SQLite data types are less powerful than PG types but it's still a database.
All of this to avoid spinning up an SQS queue?
I also question whether you really bought a horse in this analogy.
I didn't know I was trying to sell a message queue, thanks for letting me know!
I don't get it but there's a good chunk of people who are ready to jump on anything that looks even slightly comercial by even one single aspect. I've been told my personal blog was AI generated blog spam once in the past when I had literally nothing to sell. I also don't understand the fundamental issue with having a commercial involvement as long as you aren't trying to BS people or shove advertising down their throats.
It's an abstraction over a message queue, which is only one feature among many.
But btw, as a pointy-haired dude I am becoming (although I am bald), if someone really wants to buy a message queue from me, I'll happily sell it!
I don't think there is anything wrong with your article, you are quite up front about what you are selling, so personally it is no big deal.
My personal opinion is that if you don't want to come off as selling something because your genuinely interested in tech, just publish under a different domain or pseudoname
Then most of your statuses are Complete and finding an ever decreasing percentage of Incomplete rows will result in essentially a table-scan.
Then that table scan will start to take longer than your cron interval or will biff through your IO or CPU and you've got problems.
Then you might try adding indexes or beefing up your queries and then you'll realize you've just traded 6 for 12/2 because now your index updates on writes are hurting you more than your scans on reads.
Then you'll have the brilliant idea to cull your data regularly or have different tables for Complete vs Incomplete.
And then you'll want to do this culling or row-move in a transaction because you've never bothered making the job idempotent because it was implicitly so in the database all along, and now your transaction locks will compete with your cron scanning or index updates and now you'll maybe put the status table(s) in a different database and work on idempotency.
And then maybe you'll take a step back, have a nap, and wake up to realize you've just implemented a queue but not very well and have tied its design to your specific business-flow and good luck adding retry logic or new status fields.
If you're gonna use a database as a queue, make it a different database server (or schema!) from your business data. Don't tie business transactions (completing the job) to database transactions (since ultimately there will almost always be more than just one data/processing system involved), and maybe really just adding a simple Redis or SQS thing from the beginning isn't going to cost you that much so might as well squeak it in while the costs are still low.
> Then most of your statuses are Complete and finding an ever decreasing percentage of Incomplete rows will result in essentially a table-scan
This is absolutely not true. Performance of your lookups will depend on the total number of "pending" items and never on the table size.
Generally indexes on boolean columns are useless due to low cardinality, but in this specific example they'll work just fine. If you have a 100 "pending" items in a table with billions of rows, search on "pending" status will never trigger a fullscan, only a 100 lookups. You can even make your index partial, so essentially your index will be your mini-queue.
Idempotency is a valid point, but it is as hard with queues as it is with databases. Exactly-once delivery is generally not possible in distributed systems (in the real world at least, where you don't own every node). Queues won't magically solve it for you, they need to implement the same locking mechanisms as the database.
Partially this is why AWS SQS boasts to be almost infinitely scaleable, but limits you at 300 (!) rps for FIFO queues.
When I try to estimate ROI on those features it's usually miniscule. A/B test are unconvincing and the aggregate reports synthesized from the data look more like numerology than rigorous science. More-over, senior leadership usually doesn't realize how much additional liability they take on by collecting and storing so much customer data.
It's not 2010. The large tech firms are your direct competitors and regulators across the world have caught up, including in many US states. If you have a primary product for which people will pay money, then the "surveillance agency as a side-business" model of the 2000s-2010s is probably a bad idea. It's a net expense that exposes you to liability and distracts your team.
Additional benefit: without real-time analytics and personalization, you can go even further without a message queue.
That's not the architectural reason for message queues in my experience.
Primarily, a message queue is used when there is a potential for a bottleneck in the overall application throughput where there is high cost of reacting to some event so instead of reacting to the event synchronously, you queue the event and react to it asynchronously.
Some very common use cases:
1. Your UI. Operating systems use a message queue to capture and forward inputs. Your mouse and keyboard events are all queued in a message queue.
2. Webhooks. Many webhook origins have timeouts on responses and your application must respond within a given window or it will be throttled or downgraded. In this case, the best practice is to queue the event to be processed asynchronously (that queue can be a simple database table with your own logic and wrapper around it or something like AWS SQS, Azure Service Bus, or Google Pub/Sub).
3. Mitigating Throughput Bottlenecks. Most applications have a read/write asymmetry so it makes sense to optimize your architecture to scale for reads. But what if your application occasionally has to handle a burst of writes? Should you size your infrastructure for that case? One approach is to proxy the writes through a queue so you can size the infrastructure for a maximum throughput that is managed by the queue. For example, instead of 1000 concurrent writes per second, a queue can capture the write mutations and trickle out only 100 concurrent writes per second. Instead of sizing your application to scale to handle 1000 writes per second, you only need to size your queue to handle that scale.
4. Resiliency. If a message fails, it can be retried according to whatever heuristics make sense for the domain. Sure, you can use a simply loop to retry, but every message queue provides some mechanism for handling retries, failed message delivery, and so on. If you decide to roll your own and log a failed call into a database to try it again later...well, you've effectively captured a message in a custom queue.
UI - Logging mouse movement, keyboard events, and other types of attention proxies.
Webhooks and/or Bottlenecks - calling out to either internal or third party classifiers, or more recently even generative models, for personalization based on user tracking data.
And I don't think this is just my unlucky experience. The fine article includes user tracking as one of only three explicitly enumerated reasons for queuing:
> Why? Because users increasingly expect a real-time experience. In use cases like order flows, webhooks, user tracking, etc. users expect to be able to see the new data in the user interface instantly, instead of having to wait for some background batch processing to periodically reload.
Of the explicitly enumerated motivations in the article:
1. "Order flows" - tracking/modification is often one of the higher latency items in order flows ("you might also like" / "what to order next" features).
2. "Webhooks" - often used for tracking/personalization
3. "User tracking" - ...this one is easy :)
Note: though message queues are great, they make asynchronous operations and interacting with external services much more resilient.
If this were just a discussion about backend architectures or message queue architectures or Chiselstrike it might feel more cohesive.
I think it's challenging to describe simply, because you've built a system that masks over these different architectures.
Super minor things: 1) personally I find memes in technical articles distracting but I know not everyone agrees. And 2) the differing graphics backgrounds and borders on the gray background, coupled with Medium's (two) sidebars is also a little distracting.
Related possibly helpful, I get the feeling these days we write for 2 distinct set of eyes on the internet: those looking at a large screen and those who digest articles on a 3x4 inch screen. For example, one can see your diagrams and read the text discussing them on a pc at the same time.
Examples:
"But often, especially with nascent projects, we can’t afford."
"From full backend-as-a-service solutions like Firebase, to tools like ORMs to abstract and simplify dealing with databases."
"Simple, with an architecture that already internalizes all of the common issues seen when handling events."
You also sometime use sentences that are too verbose or hard to parse.
Example:
A) "In this post we’ll show how the lack of good abstractions leads to build-your-own backends lacking a key component that is present in most modern backends: a message queue."
B) "In this post we’ll explain why message queues are important to a good back-end, and why developers may overlook them. (part that doesn't need to be said: because of the focus on database-centric abstractions.)"
A) "If you ask most people, a backend is comprised of a database where the data can be stored, a business logic layer to mediate access to that data, and a couple of authentication components. But if you ask modern backend engineers, the answer differs. Yes, you have all that, but there’s a major component missing in this picture: a message queue.
From modern alternatives like Redpanda, to established systems like Kafka, message queues are an integral part of most professional backends in use today."
B) "Most modern back-end engineers will attest that a message queue, such as Kafka or RedPanda, is crucial to a good back-end."
Generally I'd get rid of the conversational, quirky tone, I think it'll improve things stylistically.
For the content itself I'd write a clearer plan with titles and subtitles, and the idea you want to convey in each paragraph.
Also a good tip is to always write the introduction at the end, it makes it clearer in your mind what you're actually trying to convey.
It's like you showed up just to dump on the author. You added no value here.
You fail to articulate why it was hard for you to read. You failed to articulate what might have been better. You failed to have a basic level of compassion and human decency in a way where you could even make a base-level attempt at wording your post as anything other than a snarky attack.
I'm personally sick and exhausted of commenters who behave like this. The people who contribute nothing but put in the extra effort to show up in comment sections to dump on people who are trying their best to share or build something.
Factually, you are the party introducing personal insults into this conversation.
i’m not sure if this pattern can be called a type of saga.
you basically have a tiny queue in you db using a special table that you drain to an external queue server.
What you might mean is that you might not need a complicated server setup e.g. Kafka for simple message queues?
it allows you to make the « add message to queue » part of a transaction that update other table which is nice!
A real queue have an API similar to JMS, ex rabbitMQ, IBM MQ, Microsoft MQ, …
The difference is you have a mailbox of message from which you remove messages you have consumed by acknowledging reception. Message will be sent to subscribers until they are acknowledged.
So publisher write message m1, m2, m3
subscriber receive m1, m2,m3 and ack only m3.
it will later receive m1 and m2 again but not m3.
Kafka is more like tcp It’s a stream.
> This works reasonably well. The polling period of reading the events may become a problem, but you can make it better by using database triggers.
to
> But when we look at tailored backends in the open, especially in architectures for FAANG, MAMAA, or whatever the acronym is until the next time Zuck decides to pivot Facebook, things look different.
unsatisfying.
Somewhere I'd love the walkthrough of in what scenarios exactly does the database-as-event-store+ad-hoc-worker architecture fall over rather than immediately jumping to what "the pros" do. That's more of an appeal to authority rather than something I can learn from.
Not saying you need to do this in your article but if not at least a link to read more would be welcome. :)
I thought of that myself to be honest but the article was becoming too long and decided to cap. Great point about the link, that's great feedback!
I've often thought there is an analogy with broken "real-world" processes where the same thing happens.
And generally this system should not use the message queue to just get things resent - because most likely in this situation you still need to be mostly handling new messages rather then old ones when you're stuck trying to catch up.
All of which tends to be a product of just utterly magical thinking from managers and solutions architects as to how anything works, and how likely an outage is (basically guaranteed no matter what anyone says about their product).
I can't find the reference, but consistent with that, I remember reading someplace that the number of items in your queue should typically be zero, since queues can't actually make the rest of the system any faster. All they can really do is buffer temporary bursts, or buffer while the rest of the system recovers. And for that you'll definitely need your recovery system working as a prerequisite.
There's sender channels, receiver channels, xmit queues, local queues, remote queues, dead letter queues, topics, subscriptions and all kinds of other things to learn about. Add that to the crappy IBM tools and their documentation that is full of dead links and you're in for a lot of learning through trial-and-error.
2nd'd & 3rd'd ...
You hired a 50 man lumberjack crew to cut down a sapling.
I've been spending a some time to remove IBM MQ out and going to "simpler" queuing solutions. First of all, cost. Guess what, we're not getting anything out of licensing per year to justify the cost.
With improving the end points, we don't need all that complexity. With 5 9's uptime on the network and smarter end points, retrying messages isn't expensive.
As always, YMMV.
Depending on your requirements you either stop the state of the whole system or use something like dead-lettering and/or delay-queues + retry count headers to make sure that you can reprocess failed messages without letting them accumulate until you're wasting CPU processing messages that will never work.
The other problems with message queues is that you need to make sure you have alerting on all your queues otherwise you're just waiting for trouble.
Would never ever architect an app in production without leaning heavily on queues because it makes so many of the hard parts trivial.
To put it in another way: if I were to build an (fairly simple, monolithic) API, would I automatically benefit from using a queue? If yes, how exactly would a queue be used, and in what way would it bring a benefit?
Queues basically serve as a decoupling layer. If you need one, you'd benefit from one.
Example one: you put incoming requests into a queue. If latency isn't a big concern, that means that your service can go down / be restarted for a moment while the queue buffers all the incoming requests.
Example two: you write a HTTP API for an interactive web application. Fast response times are important, so you do only what you absolutely must inside the request handler, and everything else that can be done later (like sending mails) is put into a queue, and can be dealt with later either by another component, or by your service running "queue worker" mode instead of in "http responder" mode.
If your services is just reading data from a DB and handing it back to the user, you usually don't benefit from a queue. But most apps tend to become more complicated with time...
They aren't _queues_ per say but they have a lot of the same DX. Depending on your runtime environment as well, you might benefit from splitting side-effect jobs from your API request/response cycle. That way you can still respond quickly while enqueueing work (locally) all still within a monolith.
Disclaimer: not affiliated with ZeroMQ, just a heavy user of BSON and ProtoBuf that goes with ZeroMQ.
In fact, ZeroMQ library (libzmq) supports over 13 different methods of queuing.
https://stackoverflow.com/questions/50656232/guaranteed-deli...
echo '{ "symbol": "AAPL", "bidPrice": 174.1 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "bidVolume": 1000 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "askPrice": 174.2 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "askVolume": 2000 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "bidPrice": 174.0 }' | rpk topic produce book-updates
However some top of the book update can result in an entire level disappearing. In that case, both price and volume changed actually at the same time, yet according to this example it will be broken down into two messages? This will lead to inconsistent state of the book when I read it between two messages. Or did the example just happen to show one field per message, and multiple fields per message are possible too?
Easy to install, sane config out of the box, developer friendly definition, relatively flat learning curve with highly googleable answers, and widespread support for AMQP.
There may be something faster/leaner or with some bells and whistles, but if you just need message queuing it's hard to beat.
Perhaps I'm colored by playing Factorio like games. It feels like managing a message queue is quite similar to managing an abstract factory line. Would be nice if I could have a similar visualization.
Getting RabbitMQ to work in a cluster is a nightmare.
Be aware of that when selecting it as your message queue :-)
If your exchanges, queues, routing keys, etc. are planned out ahead of time it should not be difficult to manage.
First of all, operating a cluster is a hassle, sometimes (for no apparent reason) on of the members gets thrown out and doesn't automatically rejoin, you have to do that manually.
Second, and this is a bigger issue: If you use a query/response pattern, the requester typically creates a temporary queue for responses, with names generated by RabbitMQ. Those response queues are only readable for the connection that created them, so even if you were to replicate those queues and messages through the cluster, that wouldn't help anybody. BUT, the service sending the response needs to know on which cluster node the response queue, and use the connection to the right node to send the answer.
(At least this is my understanding; we had an application running on a single rabbitmq node, and when trying it on a cluster, clients wouldn't get responses; maybe the reason was something else, but the fact stands that it simply didn't work).
Such response queues are also not durable, they don't survive the failure of a node.
I assume it's possible to make applications work reliably with a RabbitMQ cluster, but I guess that would need to one of the stated goals upfront. Our attempts to retrofit applications to a cluster failed.
Sorry what? How do they make so much sense? The article gives a full backstory and diagrams for event sourcing pattern it says every one knows but has no description as to why a message queue makes more sense than event sourcing?
I am so fed up of reading half way through an article just for the author to throw away their attempt at being informative and go straight into a sales pitch.
I think I am just going to say event sourcing makes so much sense and call it a day, I clearly don't need their product.
The jump from "saving events to be processed works pretty well" to "you should use message queue" wasn't explained at all.
I use redis to keep a store of last updated timestamp property (that is, the last time the data was updated), and I essentially throw out a message if the timestamp is older.
For any writing changes I make I have a validation loop in the code path as a secondary validation.
I think once you get used to been defense about incoming messages you can handle most delays
(We work with a design firm that provides illustrations for the blog)