Show HN: Build a Kafka broker from scratch, in any language
I'm excited to share our latest challenge: Build your own Kafka.
In this challenge, you'll create a Kafka broker that can read Kafka log files and respond to requests using the Kafka wire protocol. We’ve broken the project down into 18 bite-sized stages, starting with simpler tasks like implementing the Kafka protocol and progressively getting more complex. You can check out the stage breakdown here: https://app.codecrafters.io/courses/kafka/overview.
Like our other challenges, we'll continue adding extensions to this one to cover more Kafka features like transactions, replication, and more.
One of the toughest parts of building this challenge was reverse-engineering the Kafka protocol—it's notoriously under-documented, which our early testers also pointed out.
To make it easier to study Kafka's message formats, we built binspec.org, an interactive tool that lets you explore protocol formats byte by byte. Think of it like Wireshark’s analyzer, but designed for understanding protocols in detail. It visually breaks down message structures, showing each byte in context with helpful annotations. For instance, when examining a Kafka request, binspec highlights different fields—like message size or request headers—and explains their role in the protocol. Here's an example visualizer for a Kafka request: https://binspec.org/kafka-api-versions-request-v4
While working through this challenge, you'll get a deep understanding of how Kafka works under the hood, but you'll also build mastery in your language of choice. We currently support five languages—including Rust, Python, and Go—and like with our other challenges, we'll expand this based on demand.
We're keeping this challenge free in beta for a few weeks to gather feedback, so feel free to dive in and let us know what you think. You can start building here: https://app.codecrafters.io/courses/kafka/overview. Grateful for all feedback!