Distributed coordination with Apache Zookeeper | Hacker News Reader

Distributed coordination with Apache Zookeeper

igvita.com

17 pointsigrigorik7 comments

Could someone explain possible use cases?

I work on an appliance system that is really a small (<16 node) distributed system. Things like peer discovery, service lookup, leader election, distributed locks, transaction coordinators, pub-sub queues, and a reliable store for small configuration-like data are useful to our system and, I assert, most distributed systems. Zookeeper can provide all of these things.

In my view, Zookeeper's biggest shortcoming is that it doesn't support dynamic membership. That is to say that the Zookeeper nodes themselves are not self-organizing, you have to specify the ip addresses of all Zookeeper cluster members. Issue 107 [1] tracks this deficiency.

[1] https://issues.apache.org/jira/browse/ZOOKEEPER-107

In reality this tends to not be a very big issue. Typically for online production serving you want 5 ZK servers (in case you take one out for maintenance you could still suffer a failure and the quorum would be maintained). Once you setup the ensemble (cluster) it typically doesn't change much. There are some cases where this is useful, but they are pretty rare. Furthermore, you usually don't want just any process attaching to the ensemble, specifying the host/port explicitly is a security feature of sorts (granted not perfect though).

I think you're probably right about ensemble setup overhead not mattering very much if you are creating a cluster in your own datacenter for your own apps. In my case, I need to bootstrap an ensemble in someone else's datacenter without a priori knowledge of ip addresses.

I don't mean to be totally negative. I really respect what the zk team has done and it's a fantastic tool that is often and unfortunately overshadowed by Hadoop. I think a small, reliable, coordination cluster providing atomic distributed primitives is an excellent distributed system architecture. Obviously Google and Yahoo have had success with it. I just recently learned that Ceph takes a similar approach.

It can be used as a distributed lock or distributed metadata system. A lot of places hack the same functionality on top of memcache (or some other centralized store with atomic primitives).

Zookeeper feels like they spent a lot of time re-implementing Erlang's node and communication system in Java. In their design doc, they talk at length about how messages are guaranteed to be delivered in-order (and they seem very proud of this fact).

I think Zookeeper could be implemented in Erlang with 1/5th the code and 5x the reliability of what they have now.

Sounds like an interesting challenge, if you're up to it please start a discussion on the dev mailing list http://bit.ly/cpVqnb It would be interesting to see the results. Java is currently used because that's an environment that ops feels pretty comfortable with, erlang would be a great option esp if it's faster, more reliable, etc... I've heard some interest in .NET for the server as well.

The "recipes" page has some good examples: http://bit.ly/938bnt dynamic configuration, leader election, distributed queues, distributed locks, group membership, etc...

Reply on news.ycombinator.com