RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
eranstiller.com
eranstiller.com
If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option.
If you are orchestrating the concerns of 5-10 services, I think you are making your life harder than it needs to be with ESB-style abstractions. Direct method invocation is much more reliable than whatever any one of these vendors could ever sell you. Put all your services into one exe. If you can't be bothered to use one language/repo, there are still ways to achieve this.
The real architect's dilemma is shoving one's edifice-constructing ego into a box long enough to produce a useful shack for the business.
We JUST turned on remote Logs because until now Kafka didn't have capacity.
TIBCO Rendezvous was one of the first successful large scale, low latency and near real-time pub/sub implementations, and it had a very efficient, Avro like, wire level serialisation format that made messages very compact and efficient to deliver. It was very popular in finance, banking and manufacturing, and is all but legacy now.
Hard to justify a team of 7 burning several million in just Logging infra costs.
And at that scale, we need to grep the logs so the downstream need the ability to process that volume, which it couldn't until about 2 years ago.
This is a SOLVED usecase.
I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first 3 failures, as they could with RabbitMQ.
Having said that, Kafka lets you keep all your data, so you don't have to worry about losing messages to unexpected interactions between RabbitMQ rules. But having said that, now you have to store all your data.
All the smartness of the messaging can be implemented in the smart clients. Then you can expose that as a smart messaging api to dumb clients.
The most obvious example is kafka streams which exposes a "simple" api rather than dealing directly with kafka, but obviously you could create a less featurefull wrapper than that.
I pushed against knative in our company particularly for that reason. Like we wanna use kafka because [Insert kafka sales pitch], but we don't want our developers to utilize any of the kafka features. We're just gonna define the kafka client in some yaml format and have our clients handle an http request per message. It didn't make sense to me.
Its just an abstraction layer to make things less effort.
It's one thing to have an abstraction for kafka in your code, it's another to wrap the client in a smart client that reimplements something like rabbitmq, and much worse a smart service.
Im not saying to expose the same primitives - what would be the point in that? I am saying that EVERY lib you use will be using the standard lib or some abstraction of it to perform its own utility.
> It's one thing to have an abstraction for kafka in your code, it's another to wrap the client in a smart client, and much worse a smart service.
That abstraction is exactly what i am talking about. Why write 50 lines of boilerplate multiple times throughout your code when you can wrap that up in a single function call and expose THAT as your client. You know thats exactly what you will end up doing on any non-trivial project. Or you could use a lib that already does that - such as the "official" kafka streams lib.
This is decent advice, IMO. The cost of dependency management is often vastly understated.
The cost of dependency management may be understated but it’s always less than the cost of reimplementing everything found in established libraries.
Also, let's be honest: The phalanx of developers that violently argue for importing everything and never implementing anything yourselves is way bigger than people who argue for the opposite; I don't think we need to worry about the latter making things worse as much as the former.
We've seen what the world turns into in both scenarios, I would argue, and at least with the first one we got software that ran decently and developers who knew how to actually do things. We have overall much safer languages now so their home grown solutions won't have the same safety issues that they've historically had.
With the importer crowd we've gotten software that feels like molasses and an industry full of people who know only the surface of everything and more importantly become tied to frameworks and specific APIs because they never actually go beyond any surface APIs.
As with most tradeoffs in software there is a good middle-ground, but we won't ever get there if we don't have people who argue for making things yourselves.
I can imagine home grown libraries having inconsistent api with wild and wonderful assumptions and beware the edge cases
How do you do, for example, a queue with priorities client side without it being insanity? That's a relatively basic AMQP thing. Or managing the number of redeliveries for a message that's being repeatedly rejected.
You can absolutely try to build some of this with a look-aside shared data store that all clients have to depend on in order to emulate having the capability in the broker, but you just introduced another common point of failure in addition to the messaging infrastructure. Life is too short for this.
I'm just saying you can "dumb down" the client side on kafka by creating an abstraction layer (or one of the many higher level libs that already do that).
- RabbitMQ: Smart messaging, dumb client - Kafka: Dumb messaging, smart client
Have you heard of Fluvio? Fluvio: Smart messaging, Smart Client, Stateful Streaming
Kafka + Flink in Rust + WASM Git Repo - https://github.com/infinyon/fluvio
The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition.
If a certain message processing takes very long, all other messages in that partition will have to wait.
Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is much higher; but you do get nice message ordering for any given partition in kafka which you do not get in RabbitMq (afik); you are also get some really nice data locality with kafka because unless the consumers get the partitions re-assigned, all messages for the same key are served on the same physical consumer.
Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consumer.
RabbitMQ says "let me track this for you", Kafka says "you already need to track this so why duplicate the data in the cluster and complicate the protocol".
If you don't have reliable persistent storage available and insist on using the Kafka cluster to track offsets, you can track processed offsets in memory and whenever your lowest processed offset moves forward, you have your consumer commit that offset manually as part of its message loop.
If your service restarts your downstream commands need to be idempotent of course because you will reconsume messages you may have previously processed, but this would be the case regardless of Kafka or RabbitMQ unless you're using distributed transactions (yuck).
> If a certain message processing takes very long, all other messages in that partition will have to wait.
You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
And there are libraries that will manage all this for you e.g. https://github.com/line/decaton
I've seen databases used as messaging queues and if it was up to me, I'd never do that. It's usually "but we already have kafka + db, why burden ourselves with another messaging technology?", which is fair.
> You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
That is very nice -- certainly seems better than just batch processing of kafka messages, but you're still just kicking the can down the road. How large do allow the buffer to become and what do you do when it's getting too large?
You probably use a DLQ.
Don't get me wrong, I think the buffer idea probably works most of the time.
I am fond of Kafka now that I understand it, but I was also an assembly language programmer in a past life so my opinion is probably in the minority.
Regarding the buffer size: you need to implement back pressure, especially if you are CPU and not IO bound; it's another thing that's easy to get wrong with Kafka.
It combines the best of both Kafka and RabbitMQ IMO.
Looking at Wikipedia (https://en.wikipedia.org/wiki/NATS_Messaging) I see that I'm technically right, it's JetStream that does the storage layer - but it's part of the NATS server.
From memory, I really liked the philosophy of NATS but found the nomenclature confusing.
I have used it mostly for message-first services, and found subject-based messaging a breath of fresh air to decouple services. You can do the same thing with RabbitMQ topic exchanges, but it requires quite a bit more hand-waving.
So you can for example ask for [the first/the last/all] of the messages on a particular subject, or on a hierarchy of subjects by using wildcards. All the filtering is done at the server level.
You don't want to classify them by what they do. You want to classify them by what the clients must do/experience.
The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't need a lot of data to trigger a process, the message just needs to let the system know enough to locate all necessary information.
Also, when you are sending information at an extremely high rate, there usually is no difference if each message is processed separately or in batches.
So what you can do in practice?
1) Get the producer to batch the messages. For example set rules like "batch up to 10.000 messages, up to 100ms, up to 100MB of data, whichever comes first". 2) Serialise the batch (for example, if it makes sense, create a compressed JSON file) 3) Upload the file to some high throughput, scalable, cheap storage (for example S3) 4) Send a message to the queue / topic / whatever else you are using with just enough to locate and process the message -- usually just the link to the S3 object.
This usually can be modified depending on specific project needs.
Now your messaging only ever sees a small number of very small messages and you will never have any scaling problems, at least not on messaging side.
All engineering is about knowing, understanding and making tradeoffs.
In my practice, I am happy if I can get rid of hard problems (my messaging platform being unable to process X messages per second) and replace them with relatively easier problems (my persistence might sometimes fail and then I can't send a message).
I would argue that distributed persistence solutions are usually more reliable than messaging platforms and also what is a very large throughput for a messsaging solution is usually nothing much for monsters that are engineered to take much larger volumes of data. And so, in my experience, reducing load to messaging and increasing load to persistence is net positive for the overall reliability.
Messaging middleware is by design not suited well for architecting real time systems. If you require real time guarantees you would benefit from some other communication channel.
2)
3) S3 is orders of magnitude cheaper compared to messaging platforms like RabbitMQ or Kafka. If you take load off of your RabitMQ or Kafka and put it on S3 you should see a significant reduction in cost.
S3 might be more expensive than other persistence solutions, true. Just choose whatever else you have. I used S3 as an example because it is extremely easy to implement and get going.
Again, all engineering is about tradeoffs. You compare two solutions, they will always have some cons. You just decide which cons you can live with. If your platform can't process messages at all and you don't know how to scale it up to do so that's pretty large problem in my book.
To elaborate on that a bit the main things Pulsar gives you are:
1. Still underlying distributed stream based architecture, this is what makes it able to do Kafka like things.
2. Broker side management of subscription state which allow out of order acknowledgement, this means you can use it like a queue. (Subscriptions sort of act like AMQP mailboxes but without the exchange routing semantics). Vs Kafka which can only do cumulative acknowledgement, i.e head of line blocking.
3. Separated "compute" and storage. By storing data in Bookkeeper you can scale your needs to support a lot of consumers separately from how you stash the data those consumers need to read vs Kafka where these 2 are coupled and an imbalance between the two becomes awkward.
4. In built offload with transparent pass-through read. When your data falls off the retention cliff for your standard broker cluster the data can be archived to object storage. The broker can transparently handle read request for these earlier messages though, just with higher startup latency to pull the archived ledgers.
5. Way more plugability than Kafka, in-fact similar plugability as RabbitMQ. You can implement your own authz/authn, a different listener to support a different protocol (there is a Kafka one, MQTT, AMQP etc).
6. Much greater metadata scalability. Before the new KRaft implementation the layout of metadata in ZK meant that you couldn't feasibly have more than about 10k topics. Especially because of how long the downtime would be on controller failover. Pulsar can easily support much larger numbers of topics which prevents needing to use a firehose design when you would prefer individual topics per tenant/customer/whatever.
I am not a fan of Kafka, it's kinda old, and the code is a bit messy, a lot of the once only semantic problems 100% solved by Pulser are sorta kinda in Kfaka these days. All the newer stuff like built in RAFT makes it competitive with anything.
But, anyone who has used Kafka at scale has to say it 100% does what it says on the box. Many people are used to it's idiosyncrasies at scale and can get it to scale.
Now RabbitMQ. I have been burnt so badly by it breaking at scale I'll never be touching it again. Maybe my fault, maybe not, but replacing it with Kafka solved all my flakey issues and I never looked at it again.
Pulsar has mostly replaced Kafka for me because I don't need to worry about people coming along and changing requirements after the fact and saying, actually yeah we do need queue semantics. Or actually yeah we need data larger than the ~48hrs we want to store in Kafka and we don't want to teach our application to read from the archive.
Pulsar is definitely greener than Kafka in a lot of ways but the underlying stuff is very solid, BookKeeper in particular is tough so you aren't really at risk of data loss but you might run into bugs that make brokers do silly things and that can be annoying. Generally speaking though if you validate both the paths you are using and new releases it's been fairly OK to me.
The big thing was being able to directly connect external clients into Pulsar using the Websocket listener on the proxy and plugging in my own authz/authn logic. The eliminated a layer that would otherwise need to be implemented separately.
So far I have been happy with Pulsar, if you haven't tried it for a while you should give it another go. It will only get mature if people use it. :)
This is a gross oversimplification as all ELI5 are but it's a decent rule regardless.
The reason for this is that streaming systems by and large function on some sort of offset mechanism. Your client when it's receiving messages is generally calling something similar to poll(fromOffset, max) to get some messages and then keeping track of the max offset it's published somewhere (Kafka has consumer groups to help you store your offsets).
The problem with this model is you can only generally a) get messages in order on a given topic partition starting from some offset and b) you can only "commit" the latest message you processed.
This is fine if the chance of failure for a given message is the same for all messages. i.e streaming database updates into an backup. Either the backup target is available or it's not, if one message fails all would likely fail.
On the other end of the spectrum you have something like a queue of webhook jobs to execute against 3rd party/user supplied targets. The chance of any given webhook failing is entirely divorced from the rest in the queue.
So if you were to try use a stream for the webhook case you would quickly get blocked on the first bad webhook server you ran into. While with a proper queueing system you could kick that job back with a delay and process it again later without blocking work on other tasks or being able to commit which tasks have been processed.
This is generally called head of line blocking problem.
A few things they get wrong mostly about Rabbit:
+ RabbitMQ does support replay, and also has a memory only mode which will support persistance in a cluster
+ RabbitMQ doesn't have that sensitive of a latency between cluster members (no more sensitive than NATS in some setups).
+ RabbitMQ also supports Prometheus
A good (but incomplete) rule of thumb:
+ Kafka is a distributed Append-Only-Log that looks like a message bus. It allows time travel but has very simple server semantics.
+ RabbitMQ is a true message broker with all the abilities and patterns therein. It comes with more complexity.
+ NATs is primarily a straight forward streaming messaging platform.
Also consider Redis' message queue mode, zeromq, mqtt brokers (Eclipse Mosquitto) and the option of just not using a message broker/queue system. Even as someone who really likes the pubsub pattern, there's a good chance you don't need it and you may be heading to a distributed monolith antipattern.
Beauty is in the eye of the beholder, I guess. (I would definitely not consider a distributed monolith an antipattern)
Also wanted to add NSQ to your very good list.
I like the suggestion to rethink whether you actually need to be doing asynchronous computing with a message broker/queue/stream or whether you can represent your work another way.
I'd recommend considering it, especially as an alternative to RabbitMQ.
As an example, the C# API was originally very 'go-like' and written to .NET Framework, didn't take advantage of a lot of newer features... to the point a 3rd party client was able to get somewhere between 3-4x the throughput. This is now finally being rectified with a new C# client, however it wouldn't surprise me if other languages have similar pains.
I haven't tested JetStream but my general understanding is that you do have to be mindful of the different options for publishing; especially in the case of JetStream, a synchronous publish call can be relatively time consuming; it's better to async publish and (ab)use the future for flow control as needed.
Those numbers are for server 2.9.6 and .NET client 1.0.8.
I’m a big fan – never understood why it’s not at the top of the list in these tech reviews.
Are the horizontal scaling issues solved now?
it is simple to configure, has good documentation, and excellent integration into most languages. It guarantees uptime, and thats about it. It clusters really well, so you can swap out instances, or scale in/out as you need.
Rabbit can do everything Kafka does - and much more - in a more configurable manner. Kafka is highly optimized for essentially one use case and does that well. Nothing in life is free, there are trade-offs everywhere. I am not privy to which one is theoretically faster - but once you reach that question methinks the particular workload is the deciding factor.
can you be more specific? that to me sounds like hello world for either of these tools. "santize data" is an application level concern that neither rabbit or kafka would handle. as far as "pass along somewhere else" again both tools can do.
> I’ve managed a lot of rabbitmq deployments
... ?
You do need to be an expert when you start telling other people they’re wrong, though.
Nobody can really have any sources for RabbitMQ being able to do it if you don't know what it supposedly cannot do. The way you descibed it, is that you simply read data and then did something with the data and passed it to somewhere else. RabbitMQ obviously can do it.
Not true, and some also for the rest of your snotty comment, I'd have a response but it's best not to engage trolls. Another commenter answered the question I had. Good luck.
Also, did you register solely to make this comment? Pretty sad display, really.
Kafka doesnt natively support them either. That would be Kafka Connect. I guess you could use it as an MQ, but it wouldnt be a very good one. Its more used as a data integration platform. If you want more MQ-like functionality OOTB on top of Kafka you would want to use something like Kafka Streams instead.
Kafka is just a pain full stop.
Kafka itself is dumb but scalable and resilient, it's the client ecosystem that's massive compared to RabbitMQ. Count 10 stream processing, connectivity, ingestion or log harvesting platforms that use RMQ as it's backend, then name 10 languages that have supported libraries for RMQ.. then compare that to Kafka.
Traditional message brokers (RabbitMQ and similar) do support the event-driven architecture, yet the data they handle is ephemeral. Once a message has been processed, it is gone forever. Connecting a new raw data source is not an established practice and requires a technical «adapter» of sorts to be built. High concurrency levels is problematic for scenarios where the strict message processing ordering is required: the traditional message brokers do not handle it well in highly parallel scenarios out of the box.
Kafka and similar also support event-driven architectures, yet they allow the data to be processed multiple times – by existing (i.e. a data replay) and, most importantly, new or unknown at the time consumers (note: this is distinct from the data replay!). This is allows to plug existing data source(s) into a data streaming platform (Kafka) and incrementally add new data consumers and processors over the time with the datasets being available intact. This is an important distinction. Kafka and similar also improve on the strict processing order guarantee by allowing a message source (a Kafka topic) to be explicitly partitioned out and guaranteeing that the message order will be retained and enforced for a consumer group receiving messages from that partition.
To recap, traditional message brokers are a good fit for handling the ephemeral data, and data streaming platforms are a good fit for connecting data sources and allowing the data to be ingested multiple times. Both implement and support event-driven architectures in a variety of scenarios.
Ultimately, this comparison is apples vs oranges...
Never mind that the persistent durable log pattern of kakfa enables a lot replay type use cases that are very beneficial in financial systems specifically.
It’s not solely because of hype at all, it’s objectively better for many use cases.
So use RabbitMQ.
Kafka is overkill in most scenarios and you should probably just see if postgres isn't enough for your needs first (especially since you will almost certainly already need a database anyway). Kafka is more pain to setup and run than it ought to be. But underlying it is a useful and sensible abstraction for building robust distributed systems.
This seems like an important passage, drawing the crucial and long-awaited distinction between RabbitMQ and Kafka, and yet without having defined a "partitioned transaction log" the author strands the reader without any help in absorbing the distinction.
You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling properties-which doesn’t hurt at the small scale, and provides further scaling when you need it.
with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do.
> let different applications use the messages (handy for cross cutting event/notification systems)
rmq also does a publish once and fanout to multiple queues to support this. data is replicated so that could be a deal breaker, but it is possible.
how often have you had to diagnose a stuck consumer or some other kind of offset glitch where a consumer is unable to resume where it left off?
not knocking kafka here but I do think it is a tool you should reach for when you need to solve a very hyper focused problem, while rabbit is a tool more suited to most cases where queuing is required. kafka is a code smell in a lot of organizations from my experience - most do not need it.
Kafka is really nice if you don't care that much about latency during peak load and you don't have absurd processing times for messages.
Kafka had been extremely reliable with latency, even under load in my experience.
If you’ve got badly lagging consumers that are trying to read from very old points in the topic while everyone else is at the head, you’ll definitely see some increased resource usage, but again, that’s mostly a consumer issue, and I’ve need seen performance degrades that much.
I know other systems have semi-similar mechanisms, however most of them retain the “someone is the sole owner of this message” style design, which I think is fundamentally limiting. Owning application dies, is it acked or not? Acks but never gets around to putting it back on the queue? Who takes priority if 2 separate applications wish to watch the same stream of events?
I think Kafka’s “nobody owns it, acks are consumer group level” give you the same advantages for the application itself, without a number of the more difficult complications.
> rmq also does a publish once and fanout to multiple queues to support this
Which is probably fine for small volume or velocity topics, but is going to cause all sorts of load issues at higher scale.
each app would get its own queue, the messages would hit a fanout exchange that would route the same message to both queues.
Maybe 95% of cases, but not all.
Long message processing time really kills kafka in a way it doesn't kill Rabbit Mq. Combine it with inherent read paralelism being limited to the number of partitions. Add in high variability of message rates and bingo, that's like 90% of the issues I've had with kafka over the years.
yeahnah, that leads to people treating queues like databases (I'm looking at you new york times, you know what you did wrong)
its either a queue, or a pubsub, either way its ephemeral. Once its gone, it should stay gone. thats what database, object stores or filesystems are for.
Kafka is a beast, has lots of bells and whistles and grinds to a halt when you look at it funny. Yes, it can scale, but also it can just sulk.
rabbit has it's own set of problems, and frankly it's probably not choose either anymore.
You're going to have to be a tiny bit more specific here. NYT is THE factory of wrongness for sure. In every dimension. Are we talking "yellow cake" wrong, or somewhere else on the severity of f'up scale...
^ this.
All they needed was a database, or possibly a DB that supports row signing. I mean actually they could have done it with git. They don't publish that many stories an hour.
Everything about this setup is just plain wrong, and to then boast about it, absolute madness.
Currently I'm using DDS, specifically from eprosma. I would avoid that implementation unless you're using java.
I really like NATS. However I would probably use what every is bundled with the cloud system I'm using, unless its super critical.
MQTT is quite nice for things, as is rabbit.
If a consumer sees an event B, it will have certainly have seen the event A before that.
Assuming business logic is correctly written, that saves you from having to write certain retry logic on the B handlers. This requires the message queue to be always available. If it goes down, the system would not make progress (like a db - in fact the MQ is a db).
Once you add more actors/nodes to the same related events, maintaining a “causal order” can be very tricky and subtle, especially if you have an MQ and a DB as multiple sources of truth. So I’m not exactly endorsing it, even though MQ-as-a-DB (aka event sourcing) is a very interesting idea.
That doesn't fix your example of dealing with individual errors, but in many cases that's enough.
Thank you for all the feedback! Given the advances in the past few years in this area, this article could use a serious update. A comparison to NATS and Pulsar is warranted, along with some extra explanations for some of the technical terms in the article.
I'll carefully review the feedback here and try to make an update at some point soon.
1. Write a bunch of records to an S3 object. 2. Trigger a lambda to process — infinite scale out!
and if a queue really is needed due to constrained consumers, then:
1. Write a bunch of records to an S3 object. 2. The lambda trigger puts a message into an SQS queue with the S3 object's URI. 3. An auto-scaling group gets the message off the queue and processes the data in the object.
If ordering's important, then the SQS queue can be made FIFO. That has pretty low numbers in messages/second, but since records are being batched into S3 objects you can still have fairly high throughput.
It used to be that the elegance of queues for such systems would tempt developers into the operational slough of despond that is running these types of systems. Again, I'm sure it's warranted for some applications, but S3, Lambda, and SQS as above work nicely together.
What real-world cases would you use them for?
Unrelatedly, I've been looking at Pulsar lately. Anyone have experience with Pulsar and either RMQ/Kafka want to throw out some opinions from having tried both?
It also has lightweight streaming functions built-in, but they operate per record, so good for lightweight transforms/routing, not for stream aggregations etc.
It has more moving parts also, brokers are decoupled from storage, which is handled by BookKeeper, and replication between two clusters requires a ZooKeeper that both clusters can access, in addition to the ZK used by the brokers and bookies.
And it's a reasonably new project, so last time I looked into it, some of the documentation was incorrect, especially around managing bookies.
Can you do something similar? I guess, just like you can use a highlighter to paint your house. But the semantics are not correct.
- Message retention (like Kafka)
- Easy consumer scale out (like RabbitMQ)
- No particular ordering guarantees (like RabbitMQ)