We would get increased latency interacting with Zookeeper over time until eventually it would completely fail. The log messages when this would happen (either latency or failure) were extremely unhelpful. The logging server-side for Zookeeper in fact I found downright terrible.

We wound up proactively restarting the ZK cluster regularly, which improved stability.

Granted, it was our own software written to use it, and we suspect there were problems with the way it was written. It was easier for us to just rip it out than debug, however. I find it overly complicated to write against given the need for thick clients.

Consul phrases it well (https://consul.io/intro/vs/zookeeper.html):

"ZooKeeper provides ephemeral nodes which are K/V entries that are removed when a client disconnects. These are more sophisticated than a heartbeat system, but also have inherent scalability issues and add client side complexity. All clients must maintain active connections to the ZooKeeper servers, and perform keep-alives. Additionally, this requires "thick clients", which are difficult to write and often result in difficult to debug issues."

Edit: in retrospect our problems might have been solved by turning syncing off, as described here:

http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/zo...

But you can figure out from the above how scalable Zookeeper is... not very. We run in physical datacenters and I certainly wouldn't be thrilled about building out snowflake RAID systems just for our ZK clusters (we generally try to use whitebox commodity hardware and we want individual nodes to be as disposable as possible).

It's ironic that ZK requires such consistency when the goals of Mesos are exactly the opposite.