Hacker News Reader

Comment by ryanobjc | Hacker News Reader

I was interested until I saw the Zookeeper dependency.

I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

View on HN

For better or for worse, Java runs a big chunk of this world. Not running a Java application because "ewww Java is gross" will prevent you from enjoying some of the most exciting database technologies out there.

Would also be interested to hear more about the "deployment nightmares with Zookeeper". For us, it has to be one of the most stable pieces of 3rd party server software we run.

General things: when your session fails to migrate, when log compaction can't keep up with writes, when logs fill up your disk (ZK used to require external cron jobs to prune logs), when watcher notifications arrive late, when the leader's GC exceeds its heartbeat timeout, and so on.

Lots of things can and do go wrong with Zookeeper. I suspect it depends on the use case, but building a Zookeeper dependency into any system is potentially asking a lot of users/operators.

Well, like many things, if you deploy it on sensible hardware and put some thought into how its operated, zookeeper isn't particularly problematic. It's certainly not a 'nightmare'.

As an operator I do feel that pain but the alternative of every piece of software that needs distributed coordination reimplementing it, likely poorly, it isn't great either.

There are (IMHO, superior) alternatives like Etcd and Consul that could be used.

Ideally, these kinds systems should be built such that the distributed coordination piece is pluggable and the implementation can be chosen based on deployment concerns.

I have similar worries with kafka, that’s why I’ve been working on and off in my own log based message queue for while now [0]. So far is nothing but a humble beginning to something targeting very simple use cases, but AFAIK there aren't any lightweight solutions in this space.

[0] https://github.com/ninibe/netlog

Exactly this.

When I saw the header I thought to myself, "Yay! Now I can run something like Kafka, without depending on Zookeeper or having to install JVM!! Give me stable drivers for Python and Go and I am sold!"

If I have to install Zookeeper and JVM, why not use Kafka?

'plus systems software in Java', mind sharing an explanation?

Java has garbage collection and is about two times slower than C

For good C, which is hard to write - for me and other mortal humans.

On the other hand, as a mortal, I can write a-grade-above-code-that-an-idiot-would-write-just code in Java at about 10 times the speed I can write dire-useless-risible C code.

The comparison is pointless though, good modern languages like Rust and Julia are developing and LLVM is enabling further development.

You can call the comparison pointless, but it is the answer to the question of why Java is not a systems language

C dont have garbage collection. In some cases Java can outperform C.

Yeah, that's what I said. C doesn't have gc so it is a systems language. Java has gc so it is not a systems language

Java at first is great. Once you get in to large and complex loads, the GC tuning becomes load specific. While tuning GC can be fun (for the first dozen times), the bigger problem happens when your load shifts and your GC tuning becomes subtly wrong.

Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to.

Additionally, the difference between openJDK and Oracle JDK becomes something admins learn to hate you for. Complex shell scripts to invoke java, and lots of standard unix tools just don't work super well. eg: pgrep and pkill. you can tweak it, but it takes a little while to learn the many tips and tricks.

It's not all bad, most Java programs are deployed in a static-all-batteries-included fashion, so you rarely worry about system-installed library versions. So that makes deployment a little less hassle. You never have to recompile for cross platform. The profiling tooling and other stuff is pretty good, and the more you're willing to pay the better the tools get.

I find that 'jps' works as a good replacement for pgrep.

sure, but jps isnt pkill.

The point is you cant use binary name to make your way around anymore. The binary name is 'java'. Standard unix tools just don't work. Yes there are work arounds, but over time things end up being just a little more complex than they should be.

Which means if someone is comparing zookeeper and etcd, well etcd wins major points for being unixy and easy to deploy. Copy 1 binary, done. ZK loses major points here. Gotta make sure the JVM is installed, but do you need the OpenJDK or the Oracle one? if the latter, well apt-get and yum are less helpful.

It's all just little globs of annoying details that add up to be a small pain. Nothing horrible, but if you could make a choice to avoid that, why not?

Basically I guess what I'm saying is there is probably a market for replacing all the Javay distsys stuff with Rust/Go versions. I mean look at etcd!

> Basically I guess what I'm saying is there is probably a market for replacing all the Javay distsys stuff with Rust/Go versions. I mean look at etcd!

Or in the other direction. If your main system runs on the JVM and your sysadmins are used to the JVM tools then having a piece of infrastructure that's just another .jar is wonderful, and C/Rust/Go/Ruby/etc. infrastructure elicits groans. Mixing platforms will always be harder than a common platform. So the infrastructure market depends on where you think the future of applications is.

You are wrong about never getting to the 99th. I did not work on this service at Twitter, but many services at Twitter are built on the JVM, and if this service is like most there, then it probably has SLA targets that are sub-millisecond at the 99th and sub 10ms at four 9's. The only caveat to this is that it has to write to disk, which may keep it from achieving truly high performance, but this is not the fault of the JVM. I worked on one service at Twitter that had to use microseconds on its dashboards just to see any variation at all throughout the day.

Agree re: Zookeeper. It looks like DistributedLog uses BookKeeper, which I'm guessing is where the Zookeeper usage comes from.

In terms of simplifying the deployment, they could have went embedded with Atomix[1] instead. Perhaps next time.

[1]: http://atomix.io/

Mind sharing a bit more?

The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_.

So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good.

There are other operational issues people have noted above as well.

You're right in that Zookeeper can be a hassle in a dynamic environment, but in my experience Exhibitor largely removes that concern.

Then configure it using IP addresses. I've been using Zookeeper like this for 4 years in public EC2, swapping machines in and out and it always worked. What's the problem.

i think you do know the problem, which is not everyone has that set up. Yes, if you are cloud, and can roll with static IPs then great for you. Some people still have to be on-prem and reusing IPs isn't always an option.

How is this an improvement over using DNS? When you swap in a new EC2 instance it has different IP addresses from the old instance. So now not only do you have to restart your other Zookeeper servers in the cluster, you have to update their configuration then restart them.

He's probably using ENIs to decouple the IP address from the instance.

https://issues.apache.org/jira/plugins/servlet/mobile#issue/...

This behavior should be fixed in Zk 3.4.7+ and 3.5.0+

We run ZooKeeper (and Kafka) within Docker containers and were able to solve this problem with a patch to the client: https://issues.apache.org/jira/browse/ZOOKEEPER-2184

You also need to disable the JVM's insane DNS-caching behavior (config change only).

Reply on news.ycombinator.com