I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.
More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."
In a former life, I even heard this one:
- We use CentOS.
Why?
- RedHat is enterprise.
(But you’re not even paying for enterprise support)
A classic sign of "you wanted an MQ" is when a consumer writes a message to a topic to let the producer know it read the message the producer wrote...
Oof. Queued RPC is such a siren song; so many developers either stumble into this pattern or seek it out. And it's such a pain. Suddenly the latency of (often user-sensitive) operations is contingent on the latency of a queue consumer plus the time it takes to process everything in the queue before the RPC was issued. Goodbye, predictable turnaround times.
In my last two major gigs, RabbitMQ was already being run in a clustered config, and it was not going well. Both places were in the process of doing arch changes to do a change to Kafka.
It seems like something that works great in a big scaled node and you can go to big nodes these days, but I don't think it is ready for cloud/distributed durability.
I'm not aware of Jepsen testing of RabbitMQ in distributed mode for example, and I wouldn't consider any distributed/clustered product that hasn't let Jepsen embarass it yet.
Cassandra and Kafka are frequent examples of YAGNI overengineering (although the fault tolerance can be nice without scale), the reality is that pumping up single-node solutions for too long is a big trap. Projects that start to stress single-nodes (I'm thinking like a 4xlarge anything on aws) should probably get to thinking about the need for jumping to dynamodb/cassandra/bigtable/kafka/etc.
RabbitMQ --> Kafka is a pretty easy lift if your messaging has good abstractions.
relational DB --> Cassandra is a lot bigger headache because of the lack of joins.
Kafka is not operationally cheap. You probably want a person or two that understands how JVMs works, which might be something you already have plenty of, or an unfortunate proposition. But it does what is on the tin. And when you are running fleets of 3+ digits worth of instances, very few things are more important.
A distributed database will have network failures, will have conflicting writes, will have to either pick between being down if any of the network is down (CP) or you need a "hard/complex" scheme for resolving conflicts (AP). Cassandra has tombstones, cell timestamps, compaction, repair, and other annoying things. Others databases use vector clocks which is more complex and space intensive than the cell timestamps.
It's tiring to have move fast break things attitudes applied to databases. Yeah, sure your first year of your startup can have that. But your database is the first thing to formalize, because your data is your users/customers, you lose your data, you lose your users/customers. And sorry, but scaling data is hard, it's not a one or two sprint "investigate and implement". In fact, if you do that, unless you are doing a database the team has years of former experience with in admin and performance, you are doing it wrong.
"AWS/SaaS will eliminate it for me"
Hahahahaha. No it won't. It will make you life easier, but AWS DOESN'T KNOW YOUR DATA. So if something is corrupted or wrong or there is a failure, AWS might have more of the recovery options turnkeyed for you, but it doesn't know how to validate the success for your organization. It is blind trust.
AWS can provide metrics (at a cost), but it doesn't know performance or history. You will still need, if you data and volumes are any scale, how to analyze, replicate, performance test, and optimize your usage.
And here's a fun story, AWS sold its RDS as "zero downtime upgrades". Four or five years later, a major version upgrade was forced by AWS .... but it wasn't zero downtime. Yeah, it was an hour or so and they automated it as much as they could. But it was a lie. And AWS forced the upgrade, you had no choice in the matter.
Most clustering vendors don't advertise (or don't even know) what happens in the edge cases where a network failure occurs in the cluster but the writes don't propagate in the "grey state" to all nodes. Then the cluster is in a conflicted write state. What's the recovery? If you say "rerun the commit log on the out of sync nodes" you don't understand the problem, because deletes are a huge wrench in the gears of that assumption.
From my understanding of Cassandra, which kafka appears from the numerous times I've looked to be similar too with quorums and the like, it's built on a lot of the partition resilient techniques.
And, kafka has undergone Jepsen: https://aphyr.com/posts/293-jepsen-kafka
For those that don't know, aphyr will embarrass any distributed system given enough time. What is important is that
1) the distributed system is willing to subject itself to him and
2) they have a satisfactory response.
For an example of an unsatisfactory response, I give you MongoDB:
https://jepsen.io/analyses/mongodb-4.2.6
Note the "updates" section doesn't actually have them retry/repeat the testing. MongoDB just ran from the report. They claim it was fixed.
Anyway, if a system doesn't do that (submit to jepsen testing), then IMO it is hiding some big big big red flags.
Every distributed tech fails when he test it, but the tenor and nature of the report for MongoDB was different. It basically said between the lines "do not use this product".
MongoDB has a history of really crappy persistence decisions and silently failed writes, and as soon as it gets publicized saying "we fixed it in the next release". The same thing happened here of course. I simply don't trust the software or the company.
Mysql has the same annoying pattern in its history, although I have more confidence in the software because of the sheer number of users.
Still, I would probably pick PostgreSQL for both relation and document stores.
It's fair to not trust the database or company, I don't blame you for that. But I think Kyle's MongoDB 4.2.6 report was not nearly as concerning as his PostgreSQL 12.3 report which found serializability bugs in a single instance configuration, among other surprising behaviors. MongoDB's bugs were at least in a new feature in a sharded configuration. I don't think his most recent report was actually as negative as it may read to you. I say this as someone who mostly runs PostgreSQL, by the way!
As a side note I believe there are consistency bugs existing right now in both MongoDB and PostgreSQL (and MySQL and Cassandra and Cockroachdb and...) waiting to be discovered. I'm a jaded distributed systems operator :)
[1] https://jepsen.io/analyses/mongodb-3-4-0-rc3
[2] https://jepsen.io/analyses/mongodb-3-6-4
[3] https://github.com/search?q=repo%3Amongodb%2Fmongo+jepsen&ty... (note: not an expert in when or what suites it runs, just have seen it running before as a demo)
It's a complex tool that solves a complicated problem. But if you don't actually have that problem, then that's a whole lot of complexity for no gain.
Say I want to log all http requests to the server (I know I said a keyword of log) and then process those logs into aggregates, stick them in a time series.
Would it be insane to "log" everything into kafka? Or what would be the more "correct" tool for that job?
That's why that sentence in the article "but almost every technology company uses it." should be rephrased to "but almost every technology company do not need it"