KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

The current heading is ambiguous as well.

This is just an FYI type of comment. with zookeeper, one needs an odd number of machines, usually not greater than seven. Servers are indexed from 1 to n. They’re replaced one for one, so the server id matches. Scaling it up and down is a bit of a pita because every server needs to know all other servers. Changing the ensemble members requires restarting every zookeeper instance because zookeeper doesn’t support config reload.

Zookeeper does expect a single “host” for every server but it runs perfectly fine with docker. But that’s no different from consul or etcd.

Relieving to hear this as I've had almost exactly the same experience and it made me feel like an actual idiot.

Using ASGs with zookeeper is kind of a pain, but also not very necessary. Production zookeeper clusters are almost always either 3, 5, or 7 nodes and it's very rare to change that once a cluster is in production. Given that this is all very static, the easiest thing is to just use terraform or similar to create the ec2 instances. If a node dies, you re-run your terraform to create a replacement.

Alternatively, you can create an ASG-per-node so that you get auto-replacement.

In my experience, ZK is one of the easiest and most reliable distributed systems to operate. I've only seen issues when it's used as a database instead of a distributed coordination service.

ASGs + Zookeeper is really easy, but you have to use https://github.com/soabase/exhibitor. It coordinates replacing downed ZK nodes using S3, and also provides a bunch of extra management tools.

I had a single Zookeeper ASG running under load for over three years without maintenance. I pinged my old co-founder to see if he's willing to open source the CloudFormation template.

Yes, it was. More specifically Zookeeper is designed as a piece of infrastructure you put together then rely on.

Well, there's etcd, consul, etc..

edit: i guess i should RTFA -- looks like they are going to build raft into kafka -- at this point, I'd think raft libraries are fairly mature.

There are a number of Java libraries that implement Raft already -- it's a fairly well understood algorithm.

The document mentions using Raft for consensus and coordination - it's the same approach used by Etcd, Consul, Serf, RethinkDB and other systems. AFAIK it's easier to implement and understand than Zookeeper's consensus solution.

https://raft.github.io

There are other proven solutions that can be embedded such as Raft.

Yes, on both counts. However, requiring zookeeper as a dependency is also a barrier to adoption and increases the operational surface area, so offloading consensus management to zookeeper is not without trade offs.

My thoughts exactly. I don't work with Kafka, but I do heavy work with Solr that also has ZK as a dependency.

To end users of whatever platform it is keeping together, Zookeeper is often seen as an unwanted dependency. The unwanted sentiment arises because maintaining zookeeper is not a gimme, and it takes additional knowledge and overhead to maintain. You need a quorum to be kept alive with an odd number (greater than one) of instances. So the question often arises, "how can we get rid of zookeeper?".

Not to mention that ZooKeeper passed Jepsen (on the first try IIRC). Don't replace something so extraordinarily dependable just because it's a dependency.

As an operator, having fewer dependencies I have to deploy and worry about is a boon to my productivity.

If you were building a kafka like system in house for private use only, then yes I agree with your sentiment. If you are building something to be used by hundreds or thousands of organizations then the cost benefit tradeoffs shift to where it probably makes sense to pull the consensus logic into the primary application itself.

Finally somebody said it. Yes it is hard. Zookeeper is time tested and I don't think there is anything revolutionary to keep state less than 2n nodes. Either Kafka will compromise on consistency or give a solution which is as performant as Zookeeper. May be they are looking for something which is more specific to Kafka and can be done in a much easier algorithm than Zab?

Reading the proposal, it seems like they want to replace it because the ZooKeeper API is causing some issues for them, so even if ZooKeeper has perfect consensus, Kafka itself won't necessarily. From the "Metadata as an Event Log" section

> although ZooKeeper is the store of record, the state in ZooKeeper often doesn't match the state that is held in memory in the controller

Maybe it would be better to put the effort in to changing ZooKeeper instead of writing their own consensus system in Kafka, but I would assume they know that tradeoff better than us since they work so closely with ZooKeeper.

No, contrary to the sibling comments I would disagree that consensus is difficult. The theory can be somewhat daunting, and ZooKeeper's consensus protocol is complex. The most advanced consensus protocol today, Paxos (and its variations), is notoriously hard to understand.

But then we got the Raft paper, which has single-handedly essentially managed to commoditize distributed consensus. Raft is an elegant and simple protocol that has been implemented in many languages. To implement primitives like leader election and consistent, distributed reads/writes, you can grab an off-the-shelf library like Hashicorp's Raft library, which does all the heavy lifting as long as you implement the state storage yourself. It's absurdly trivial to turn a single-node app into a replicated, fully redundant one.

Of course, a project like Kafka might have particular requirements (I wouldn't know) that would mean Raft would not be a suitable solution for them, or would require modifications to the protocol. CockroachDB, for example, had to modify Raft ("MultiRaft") in order to achieve the scalability they needed within Raft's paradigm. Raft is a starting point more than a finished solution.

Exactly, and tbh this kind of thing pisses me off. The "Kafka people" seem to imagine that what Zookeeper does is dead simple and they can bash together their own in a weekend and it'll be better.

Zookeeper is great. Weird, for sure, and a bit of a pain in the arse. But it will neither lie to you nor lose data and this is much, much harder to achieve than you might think.

Where do you see performance comparisons to Kafka? As far as I can tell, the product you linked doesn't exist outside of a blog, so this reads like a marketing post.

First, because it's yet another dependency. Consensus-based systems like CockroachDB, Dgraph, Cassandra, Riak, Elasticsearch, ActorDB, Rqlite, Aerospike, YugaByte, etc. are wonderfully easy to deploy because they can run with no external dependencies. (Their consensus protocols may have different problems, but that's beside the point.)

Secondly, it's a pretty heavy dependency. The JVM is RAM-hungry and it's difficult to ensure that it always has enough RAM. Running multiple JVM apps on a single node must be done carefully to make sure each app has enough headroom. It consumes considerably more RAM than Etcd and Consul.

Thirdly, I think it's fair to say that ZK is showing its age. It's notorious for being hard to manage (see the other comments in this thread), with a fairly old design (based on the now-ancient Google Chubby paper) that, while resilient, is also less flexible than some other competing designs.

The author now works for Confluent.

Not to take away from what you did but the Apache Foundation actually has Ratis which is a Raft implementation: http://ratis.incubator.apache.org/

Care to elaborate?