how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?
Right now my preference varies a bit depending on the rest of the tech stack, but for the most part I use Redis or RabbitMQ.
If the stack is already hard dependent on AWS or another cloud, then SQS or whatever is also fine.
I also wouldn't overlook just using your existing DB (like postgres)! At low and even medium scale this can be totally fine, and also comes with lots of benefits like single-source-of-truth, normal relational DB constraints, transaction wrapping, and more. One of the highest scale apps I've worked on uses Postgres for queueing. It's take a number of optimizations over the years as performance starts to fall due to scale, but it's doable.
And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea.
There were two basic issues.
The first is that because of the nature of message queues, they're either empty or full. The second is that for many of the ways that queue are used, the unit putting the request on the queue in the first place may well be waiting for the response related to the request. So you've just turned an every day synchronous request into a more complicated, out of band, multi-party asynchronous request.
If your queues are not empty, they they are filling up. And queue capacity is a two fold problem. One, is that you simply run out of space. But, more likely, referring to the earlier point about waiting for a response, is that you run out of time. The response does not return fast enough to manage your response window to the unit making the request.
This is a load balancing problem. If the queue is filling you simply don't have the capacity to handle the current traffic. It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue.
The second part is that if you're throwing items onto a message queue, and you "don't care" about them, that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table? If the request is Important, you certainly don't want to trust it to a message queue, a device not really designed for the storage of messages. Messages in a queue are kind of trapped in no mans land, where the easiest way to get to a message is to dig through the ones piled in front of it.
They're interesting insights and worth scratching your chin and going "Hmmm" over.
Yeah, this is simpler for the requester, but not for the counterpart that has to respond. Because now, the responder has to have 100% uptime and better not fail during the request, otherwise things get lost.
Let's take sending emails as an example. You have a server A that can send emails, you have a server B that fulfills requests/actions by a user. Let's just assume that this is the (legacy) setting we are dealing with.
Now, what do you do if you want to send the user an email on a certain action, e.g. a password reset or changing a payment information etc.? Is B then making a synchronous request to A? What if server A is currently down due to maintenance? What if A uses another entity to send the emails, which itself is down? How do you observe the current state of the system and e.g. detect that A is overloaded?
With a message queue you get all those things for free and A doesn't even have to have any persistence layer for "retries" in case of crashes etc.
While it's true that those issues can all be resolved by "by load balancing or persistence" it just means that you now traded one issue (having a message queue) for multiple issues (having database(s), having load balancer(s) and essentially re-implementing a part of a message queue).
In most cases a message queue seems like a good trade-off.
But even if I've handed off identity management entirely, I almost certainly do have some per-user state. Otherwise... what on earth am I doing with users in the first place?
But okay, fair point - password-reset is maybe not the greatest example. But it doesn't invalidate the general point (I gave a couple of other examples).
> Otherwise... what on earth am I doing with users in the first place?
Maybe just making sure that the user is known and has paid for plan X (if the 3rd party service offers that).
> The first is that because of the nature of message queues, they're either empty or full.
... Wat?
> that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table?
Performance is why. The fire and forget aspect is like udp in the sense that you don't need to ensure ordering of messages (packets) or hard persistence to the database. Also, dead letter queues exist for a reason.
Message queues are super useful. The highest performance systems I've seen use the message queue + pool of workers paradigm, as it allows you to better smooth your load (unlike immediate republishing like in sns, which requires hardware available to accept) with minimal guarantees (unlike a write ahead log such as kinesis). The buffer is also great because it allows you a bit more time to scale up both your message queue fleet and worker fleet when you get a load spike.
> ... Wat?
I interpreted it as "they are either trending towards empty or full". The statement doesn't seem well thought through.
That might be true (either empty or full) most of the time (maybe) _if you squint_, but the entire point of the Message Queue is to provide buffering from the transient state (somewhere between empty and full) trending toward the empty state.
Longer SLAs mean it's also easier to hit those SLAs. Giving the clients realistic SLAs is super important.
That's false. Virtually all MQ systems are designed to persist (often with replication/redundancy) and store data. Most MQs also support non-persistent delivery, with the cost/benefit (ephemerality/performance) that entails, but that doesn't mean that durable storage is any less well-supported.
Sure, folks have plenty of operational war stories regarding failures of persistence in their MQ broker/cluster/whatnot. Same as the DBAs who manage relational databases.
Pulsar is a Kafkaesque system that can act as distributed log _or_ an MQ.
All of those are fine is you only need pub/sub
Rabbit's defaults are still unfortunate, in my opinion: queues and messages are not disk-persisted by default, though this can easily be enabled. As a result, many folks run and benchmark a "high availability rabbit" only to discover that they're benchmarking distributed state stored in memory, not disk.