> Zookeeper (
http://zookeeper.apache.org) is much more fully featured and mature, but felt way too heavy compared to my nimble Go stack. Installing and maintaining a JVM just for Zookeeper made me uncomfortable.
I've spent almost a year dealing with a large, high-traffic zookeeper installation and I agree. Well, actually etcd seems to have some features which I would love to have in Zookeeper.
I maintain several Zookeeper ensembles which I want to be highly available. Any time I need to swap out a node, increase or decrease the size of the quorum, change a node's port(s), or change a node between voting and observing, I have to draw out a diagram where I keep track of which nodes "know" what at which times. If I skip doing that, I run into situations where fewer than N/2 + 1 nodes agree on the current state of the ensemble, and they fail-stop and don't serve traffic.
Here's a specific example of an issue that seems so blindingly obvious to me, but it clearly wasn't to whoever implemented it: if you want to specify that a zookeeper node is an observer (doesn't participate in leader elections), you have to put that in the config file in two places: once on the line for that node in the section where you tell all the nodes where all the other nodes are (like [0]), and you also have to have a separate line "peerType=observer". This last bit means you can't use the same zoo.cfg file for your observer nodes and your voter nodes, you have to keep two zoo.cfgs, make your init script or whatever use the correct one, and keep the files semantically in sync if you ever have to make further changes. What they should do is have each node look at [0] and say "oh I'm server.4 so I'm supposed to be an observer".
It's piles and piles of little annoyances like that make me dislike Zookeeper. I'll be watching etcd.
[0] zoo.cfg
[snip]
server.1=hostname1:port1:port2
server.2=hostname2:port3:port4
server.3=hostname3:port5:port6
server.4=hostname4:port7:port8:observer