MQTT vs. Kafka: An IoT Advocate's Perspective

From what I understand it's generally The Thing To Use when you have not very many (ie: thousands) high-volume sources that have good links. You push stuff into the logs and you can take your time chewing through the log.

MQTT was designed for shitloads (hundreds of thousands+) of small devices on bad links connecting to the mothership.

Neither of them really are a message queue. I mean, they queue messages, but they have different goals than, say, ActiveMQ or RabbitMQ (which both can be used as a backend for MQTT).

Using Kafka for a message queue is overkill. It's more likely that MQTT fits your bill.

Lots to unpack here. The simple answer is: you use Kafka when you need to have the data in order in arrival written to disk but you don't need ad-hoc query. Messages must be persisted, processed at least once by someone, in order. It's common to have Kafka in front of a database as a sort of buffer: when data is in Kafka, it has been acknowledged as seen and will be processed later.

Now, why would you store it in Kafka rather than in a table with timestamps. I said in my previous comment that messages are persisted based on the arrival time on the broker. This is somewhat inaccurate. Kafka is not related in a message arrival time. It's interested in message arrival. All messages arriving on a partition are process in order of arrival. Messages are stored on disk as byes, each message is written to a segment file, a partition will often have more than one segment file. Segments are sequential. A message is written at a byte offset within a segment and the format is something like: {metadata, length, message body}, the protocol is documented here: https://kafka.apache.org/protocol. Each segment file has an index file. The index file maps the partition message offset to a byte offset within a segment file. To read a message, the broker needs to know the message sequence you want. The broker will identify the segment by looking at (AFAIR) the index file name which contains the first message offset within its respective segment. It will read the index, find the byte offset of the message, check the message length, read the message, and return it.

However, as you can imagine, this is a pretty involved scenario because it implies a bunch of random reads from disk. The point of Kafka is sequential reads over complete partitions or segments. So Kafka shines when you need to read large sequences from a partition. It will identify a segment, memory map it and do a sequential read over a large file (usually a segment is something like 1GB).

This is different than inserting into an SQL table with timestamps. First of all - you will have duplicated timestamps. The only way to avoid duplication is auto-incremental indexes. These will be similar to the message offset. However, a large number of writers will inevitably lead to a congestion on your auto-incremented sequences. Next, your table will have indexes: as you want ACID, those indexes will have to be updated before data is considered stored (C). Next, you have the data stored but it's an SQL database, so your data is on one node. This is different than Kafka minimum replication factor where the data is considered as stored when at least N replicas confirm it. As for reads: so your client application would store it's own last processed sequence number from your auto-incremented index, and this would work the same as the consumer offset in Kafka. But it you have hundreds of GBs of data on your server, you'll be hitting disk pretty often for index scans, or your indexes will outgrow your RAM. Sure, you can scale to a bigger box (costly) or you go for a DRDBMS (YBDB or Cockroach or similar) but due to how your tables are distributed in those systems, you will hit a latency cliff. Reads are usually better than writes, writes can be atrocious, DRDBMS large batch writes can be 500x+ slower than single node RDBMS.

> so you can replay them/fetch them if needed?

If you need ad-hoc queries, that's the best way to go. But for volumes of data larger than your SQL server, I'd consider Kafka in front and SQL to store only what needs to be queried.

> Why do you need a sharded Kafka cluster?

Three reasons:

1. Write performance. I wrote higher that messages are written in order into a sequence file. I have personally worked with clusters ingesting 400Mbps per partition on multiple partitions on a single node. Basically, Kafka can saturate your bandwidth and if you have SSDs, everything what's ingested, will be persisted because Kafka uses zero-copy and memory mapped files. So it's basically like writing to RAM. If you need to ingest more data than a single node can handle, you need more machines.

2. Read performance: data within a topic partition is written in sequence and will be read in sequence. Your typical scenario if using some derivative of a key to establish the partition number. By default kafka uses murmur2 hash. What this basically means. Imagine that you have data for email addresses. The email address is the key. All data for a given email address ends up on the same partition. Thus your consumer will always read from one partition for a given email address. The more partitions you have, the higher is the number of consumers. Of course one there will be multiple email addresses within a single partition because their hash maps to the same partition number. The higher the number of partitions, the higher the granularity of that distribution. However, it's easy to hit a hotspot for a key if you have a lot of data for a hot key.

3. I would argue, the most important one: replication. If you have 3 brokers and your topic has a replication factor of 3, you have one copy of data per physical machine, all data consistent across brokers. That is - each partition and every segment exists in three copies, one copy per physical broker. There will be multiple partitions for the same topic on the same broker and each will be replicated to other brokers.

Taking a broker out does not impact your operations. You can still read and write, you will have under-replicated partitions but Kafka will automatically recover once the missing broker comes back. If a broker falls out, and that broker was a leader for a partition, Kafka will elect a new leader from one of the remaining brokers and your consumers and producers will automatically continue writing like nothing happened.

> Most businesses are going to have Redis, SQL, and probably RabbitMQ.

> Where/why add Kafka to that stack?

Kafka is an all-purpose strictly-ordered-within-partition buffer of data with automated replication and leader election. Multiple consumers can read from the same topic partitions at the same time. Each one has its own offset and can consume at its own pace. Each consumer can do different things with those messages at the same time. Those consumers can be located at different org levels. Kafka is great at moving large volumes of data within the organization and facilitating other processes, for example ETL.

Man, I sound like I'm some Confluent salesman. I'm not affiliated. I work for Klarrio, we build streaming systems.

When the use case warrants it and when downstream consumers need to be notified of a change in the data or its state. That is, in reactive architectures.

A data streaming platform (of which Kafka is one example) will push the data, whereas a database (relational or not) requires to be explicitly polled (queried)[1]. A poll for new data is inefficient, is computationally expensive (a waste of CPU cycles when the data has not changed) and is prone to create creating delays in the data processing (when the polling interval is too long). Data streaming platforms will deliver the changes in near realtime.

Also, Kafka, being a distributed append only log, keeps an entire temporal history of all data changes. That is, if an event or an object has undergone a series of state transitions, such as create -> update 1 -> update 2 -> update 3 -> delete, it will be fully available in a topic as five separate events. If a consumer is only interested in the latest state change, a series of events can be compacted into an equivalent of a database table (naturally, it is called a K-table in Kafka) and the consumer can use the K-table instead of having to process the entire event stream[2]. K-tables and event streams can be flipped over with no extra effort.

Kafka is, effectively, your databases's own transaction log sans the query engine.

> Why do you need a sharded Kafka cluster?

Kafka cluster are not sharded. At least not in the database sense, as it is a cluster. A consumer sees one living thing[3] – the cluster, and the cluster can be dynamically resized without the consumer(s) knowing or noticing it. The cluster will then rebalance itself.

> Where/why add Kafka to that stack?

Kafka is typically used to get the data out of data sources and deliver a temporal series of changes to interested consumers. If the data needs to queried ad-hoc or explicitly, Kafka is then fronted by one or many (usually, domain specific) event stores. An event store can be a database (a RDBMS or a document one), a document search engine (e.g. Elastic Search or a vector database) or anything else that has a query engine in it. It can be a cache as well when either the speed of retrieval or affinity to the data are important.

[1] It is possible to mimic the Kafka temporal history via the use of triggers that would inject a copy of the row being changed into an «audit» table before allowing a update to occur, but, with time and in high data volume environments, the table size will balloon and will drag the database down.

[2] Most databases now have change event streams that abstract the database transaction log away and deliver raw row changes as a temporal history of events, which is a functional equivalent of data/event streaming (using Kafka or similar) sans the scalability and the distributed processing – the onus to implement either or both is entirely on the consumer and that part is hard. Therefore, a database change event stream is oftentimes hooked directly into a Kafka cluster.

[3] Not entire true in most implementations, but it is rabbit hole for another time to go down into.

You realize a time series database is more appropriate for that?

We use NSQ extensively and its a great fit for us. It's fast, lightweight, very easy to understand and work with, has great HTTP APIs for controlling the queues and publishing messages, etc. What it doesn't have is replication, so if a message must not be lost ever I'd reach for a distributed log, probably looking at Redpanda before Kafka, and possibly building my own on top of ScyllaDB since we're already running that.

Personally, I do not find Kafka difficult to operate. ZooKeeper is basically a boring black box and there’s also kraft (the famous KIP-500).

Until a manager comes on and decides it does and you have to redesign everything related to the message queue.