Hacker News Reader

Comment by elric | Hacker News Reader

The author does not mention the software in question. My assumption is that it's ZooKeeper. Simply because in my previous role I had to build the exact same thing. We didn't have thousands of nodes, but we deployed (and managed) ensembles on prem at customers. Things would go tits up quite regularly, and would be difficult to recover (or even identify). Netcatting 4 letter magic words to ZooKeeper, parsing and aggregating the results was how we got a handle on things.

Eventually we built this into the application we were deploying, so that it could monitor its ZooKeepers, raise alarms, and give us insights into aggregated reliability data.

View on HN

Pretty sure it's about Google's Chubby. It's my understanding Zookeeper was largely inspired by Chubby.

Concur. Zookeeper was absolutely inspired by Chubby, and this story fits in every detail with how things worked at Google back in the day. (Source: I was at Google from 2006 - 2010. Also, I believe the origins of Zookeeper are a matter of public record, though I'm too lazy to check right now.)

What does zookeeper even do? I know it's supposed to be a key value store. But the only thing i ever hear about zookeeper is not people actually doing stuff with it, but rather doing stuff for it.

It seems like the most useless piece of software. Like someone wrote a concept down but said it was software.

There are good use cases for it. If you have a cluster (or clusters) with many, many nodes running your application you need to be able to communicate changes/configurations to those nodes/applications. Zookeeper is really good at doing this because it itself is clustered (HA, redundant). Updating all of those nodes/applications manually is not a desirable thing, so you have the application call the zookeeper service to get those configuration changes.

The problem I see often is that it's wedged into places for which it's absolutely overkill. As in clusters with less than a dozen nodes, no service level determining such a need, etc. - it's often just a shiny object for architects.

> you have the application call the zookeeper service to get those configuration changes.

as a note, both ZK and etcd have key watches such that one need not poll the key-value store, it'll push out notifications for changes to subscribers, making for super quick reactions in any such distributed applications

It basically offers a set of distributed primitives which lets you build distributed applications with far fewer headaches. To be fair, it's mostly pretty good at what it does, but it's not super ops-friendly, and when things do go wrong, recovery isn't a whole lot of fun.

Apache Curator builds on top of ZooKeeper and turns the primitives into usable recipes, which is what I recommend to most people these days.

Like, I'm a consul user, but I'd assume the use cases are similar.

One concrete example: Consul can provide you with locking and thus leader election with relatively little effort. Patroni, the postgres manager, can use a consul lock on a key to elect the postgres leader and consul ensures mutual exclusion.

Or in a similar way, I needed to run some data collection once a day, but the cluster consisted of 3 identical nodes and I didn't want to start treating one node specially.

I could use consul to hold a lock by using <consul lock whatever/lock collect-data.sh>. Once I had mutual exclusion, I could store the last time data was collected in a consul kv entry with <consul kv put whatever/last-run $( date +"%s")>. Then each script just grabbed the last time data was collected or 0 if missing and checked if it was 23ish hours ago. And there we go, all systems identical and fault tolerant once-a-day data collection without additional config to forget on top.

I've thus far avoided using it, but my recollection is that it kinda snuck into mainstream stacks because it was part of the standard deployment model for the ~2015 era of "big data," with Hadoop, HBase, Solr, and the other usual suspects from the Apache Java ecosystem.

Nobody chose to have Zookeeper in their stack, but they needed it for these other services and so eventually they were stuck with it.

Reply on news.ycombinator.com