Kafka in Pieces
thebaffler.com
thebaffler.com
(currently I'm rewriting engine and when I'm done, I spent time on something easier to do than Castle, probably Slayers)
It certainly feels like the people who run the world these days read a lot of Kafka... What reads like a horror story to one person reads like a fairy tale to another.
WHILE THE PEOPLE SLEEP, THE STATE DREAMS
Come for the distributed, replicated, partitioned log. Stay for the existential angst, sinister bureaucracy, and paranoia in the face of the absurd.
It's good if you have a use for it.
Internally it's one of the simpler pieces of software I rely on. It's base API is small and effective and works "as you would expect". Replication uses the same fetch API consumers use, because why wouldn't it? Controller election uses tried and true ZK patterns, it's log storage is pretty simple, even the compaction logic is understandable.
To be fair I have spent a lot of time with it, patched it and it's pretty much the core of my toolbox but I don't think this is a controversial opinion among data infrastructure engineers. Compared to other software we work with Kafka is some of the simpler, dumber stuff which is refreshing.
On the other hand I really want to learn Pulsar but it definitely is more complex, my hope is that complexity pays off with big architectural advantages but we will see. :)
On the topic of what to use it for chances are if you don't know what it's for you probably don't have the problems it's meant to solve. I don't mean that in a derogatory way just that it's designed for large distributed architectures where many interested parties want to consume the same data. Or where a small number of very high throughput applications need a buffered transport that can take the load and spill to disk etc. I.e it is inherently niche, most companies don't have these problems.
I know, respecting the original title is a nice thing. On the other hand, HN is a tech oriented web site and +90% of the times if there is a Kafka in the title it is related to the stream processing framework, not the writer. So, just writing the writer's full name (in those rare occurances where the writer is mentioned here in this news site) can solve that tiny problem.
Those are my two cents...
> +90% of the times if there is a Kafka in the title it is related to the stream processing framework, not the writer
This makes me think why these crappily implemented software choose such pretentious literary names. 'Linkedin Sprof' would be a suitable name for this below average software.
People often ask how Kafka got its name and if it signifies anything specific about the application itself. Jay Kreps offered the following insight:
I thought that since Kafka was a system optimized for writing, using a writer’s name would make sense. I had taken a lot of lit classes in college and liked Franz Kafka. Plus the name sounded cool for an open source project.
So basically there is not much of a relationship.