Hacker News Reader

Comment by antirez | Hacker News Reader

Here the point is more no Zookeeper needed apparently.

First, I'm a big fan of yours. I was recently interviewed by Go Time and I talked about how awesome Redis is, what a good example it/you've been for open source projects, and how funny your v4 RC blog post was.

My goals for building Jocko to start:

1. Learn more about distributed systems to the point I could build one "What I cannot build, I do not understand." sort of thing. 2. Simplify running Kafka—so remove ZK, only need to install a binary, and so on 3. Maintain compatability—so people can just drop Jocko in, Kafka clients and anything else that uses Kafka's protocol works

After that, I have some ideas on things to do but am waiting to get feedback from people. For instance, I think the configuration around setting how much disk space topics use could be better.

Hello, thanks for the kind words! And sorry for guessing what your goal was, I used the wrong words probably, what I meant is that for me the highest value of Jocko at a first glance is the tremendous operational simplification provided by the fact it does not need ZK while implementing the same protocol. Of course I guess the project is currently ongoing, but once it will be a drop in replacement for the Java implementation needing ZK this could be very interesting for many users. This project could also serve to Kafka developers as an inspiration to also remove ZK perhaps, I guess that they already reasoned towards this idea tons of times btw so there must be some rationale they have (based on their design choices) to keep it.

Having distributed consensus as a distinct service (as opposed to directly building it into each application) is beneficial to a lot of companies because they can invest resources to make it operate well (in addition to being correct) and reap the benefits across all of their services. By well, I both mean having competent operations around the service as well as making it more resilient and faster to respond to various networking failures.

the downside is that in order to use something like kafka locally (afaik?) you have to run zookeeper locally as well.

Also, speaking with my sysadmin hat here, Zookeeper is an insanely powerful introspection point for running distributed services. The number of services with internal cluster coordination protocols that offer similar capabilities is abysmal.

There are other service discovery tools than Zookeeper such as Consul.io, Etcd, etc.

The biggest benefit IMHO would be from making this a pluggable service within the Kafka client and server. Then, you're not tied into ZooKeeper explicitly.

Reply on news.ycombinator.com