Topicctl – an easier way to manage Kafka topics | Hacker News Reader

Topicctl – an easier way to manage Kafka topics

segment.com

84 pointsmrbbk15 comments

Segment employee here. Our Kafka runbook looks 100X cleaner after this tool was rolled out by Ben and folks. One of my favorite parts is how it makes dangerous things harder :)

nice! Just skimmed the docs. Does it handle ACL or stretched cluster yet? Planned?

Not yet! But would be happy to look into those and/or get help adding them.

Did you look at using terraform for managing kafka topics?

Segment employee here.

Yes, we looked at Terraform to manage our topics. We are big Terraform user to manage our infrastructure. We mostly didn't go that way because we didn't want to have to manage the Terraform state. We also wanted the tool to be extensible and writing custom TF providers can sometime be tricky.

Is there a reason you chose not to use ACLs? We did enable them, but it feels like it is causing more headaches than any security benefit it might provide...

I guess we just haven't got the need to use ACL yet. This may be something we will look later on. We have been started to implement what's needed to leverage things like SASL in our instrumentation but we are not there yet (see https://github.com/segmentio/kafka-go).

Thanks for this! Managing Kafka and topics can be messy at times, so a really welcome addition to apply more robust Kafka changes.

> Replica placement strategy

This is huge, currently you have to manually specify placement for all replicas when you increase the replication factor for a topic. Do this for many topics and it quickly becomes very tedious and error-prone. Thanks for this, I'm very excited to try it out!

"""Many of these were shaped by our experiences making AWS changes with Terraform and Kubernetes changes with kubectl. We wanted an equivalent to these for Kafka!"""

so use this topicctl for advanced topic changes.... and the more primitive kafkactl[1] for basic stuff?

What about the 10 or so other GUI managers like this one? https://github.com/cloudhut/kowl [2]

[1]https://github.com/deviceinsight/kafkactl

[2] https://news.ycombinator.com/item?id=24099037

Author here, thanks for the comment. topicctl is really motivated by a desire to support rigorous, git-based topic management. The read-only views (tailing, repl, etc.) are secondary to that and are useful for our command-line-based workflows inside Segment, but are definitely not intended to replace all of the other good tooling out there, including the ones you've referenced.

I think the key part is that your topic config is stored in version control.

kafka-gitops [1] is another declarative topic/acl management tool that enables a planfile/apply strategy.

[1]: https://github.com/devshawn/kafka-gitops

That looks very interesting! About managing YAMLs like we'd do with kubectl as explained in the article, how does topicctl differ both in concept and operation from Strimzi's Kafka Operator for Kubernetes? I know, they are a different company etc but that Kafka operator is gitops-friendly and everything so I'm honestly trying to compare them as managing Kafka is a REALLY HARD problem, so kudos for trying to make it easier!

At Segment we manage clusters which are both on ECS and EKS. We didn't want to tie our Kafka management to the container orchestration.

If I'm right the Strimzi operator is also managing the brokers. Here we only focus on the topics.

Reply on news.ycombinator.com