Building Self-driving Kafka clusters using open source components | Hacker News Reader

Building Self-driving Kafka clusters using open source components

slack.engineering

45 pointskiyanwang18 comments

I hate Kafka with a passion. We’re driving it with Kafka Streams, and our Streams threads have been blowing up left and right with undecipherable error messages whenever we look at it slightly wrong. We also don’t have the resources to babysit it hourly so here we are, plagued with undebuggable production downtime and no way out of this.

I’ve been writing software for a living for more than two decades, and Kafka is the most horrible, impossible-to-grasp boat anchor I’ve ever had to deal with.

I hate every minute of my working day that I have to deal with Kafka Streams’s hissy fits.

You're conflating Apache Kafka, the distributed log and Kafka Streams, the streaming library built by Confluent that leverages Apache Kafka.

Kafka itself is straightforward to grasp, Kafka Streams, yep, your concerns sound familiar.

Kafka Streams is a core part of the open-source Apache Kafka project.

You may be mistaking it for kSQL, or ksqlDB as they call it now, which is a Confluent project built on top of Kafka Streams.

Be that as it may, I agree with the spirit of the grand-parent comment, which is that if you stick to straightforward Pub/Sub Kafka works great.

It is, but it's ancillary to Kafka the distributed log.

Same experience here. We're using Kafka Streams and it's a horrible experience due to its unreliability. We need a full-time person just to make sure it stays up.

Same experience here. The only thing excellent about Kafka is marketing. It is endless, relentless and everywhere.

Have you tried nats.io and it’s new streaming tech, Jetstream? Night and day difference in terms of management and ease of use. In my last org I was fighting so hard against folks wanting to adopt Kafka, and then I found out about nats, total game changer

Or Redpanda. We like to think it's Kafka but better.

Obvious bias: I work for Redpanda

NATS is open source and CNCF. Redpanda is more a SaaS -- if it goes away, so does your app.

I don’t think that holds here, Redpanda has a Kafka compatible API layer and you can 100% self-host it. Being CNCF or not doesn’t seem like a meaningful or useful distinction here.

The compatibility isn't 100% there yet.

https://github.com/redpanda-data/redpanda

You're welcome to use it yourself and not pay us a dime. We're a business so we need to monetize but our core is to build a better Kafka. The nice thing is that if you've already invested in building against Kafka but you're exhausted by running Kafka, ours works better and faster. (Again, I'm biased so don't trust me, verify.)

NATS and Kafka offer different semantics.

By default, NATS offers non durable messages and different delivery options. I believe you could get durable messages, but last time I read up on this, it required running extra features or infrastructure to support this.

You now get durability on whatever subjects you want via NATS Jetstream: https://docs.nats.io/nats-concepts/jetstream

No additional infrastructure, just enable it on the nodes you want persistence on and create a stream. Publishers don't even have to change, consumers have a great api with lots of options

Oh that’s pretty cool, I wasn’t aware of that! Nice to that functionality added!

I'm confused about the breezy way in which they distribute partitions and consumers. Since the producer computes the partition key and the consumer chooses which partitions to bind, how does the central kafka team enforce balance?

One would need some sort of policy about how developers use the Producer and Consumer API's for these things, but that would be a very difficult thing to manage and enforce I would think.

Their upgrade strategy is really awful. Kafka can do rolling upgrades, it's one of the few things that has worked reliably well for me in Kafka over the past 6 tears. Pretty much as simple as a broker reboot and setting inter.broker.protocol.version/log.message.format.version correctly.

They made it way harder than it needs to be.

Reply on news.ycombinator.com