Hacker News Reader

Comment by tbragin | Hacker News Reader

Coincidentally, as someone who worked on this blog, I was surprised (and pleased!) to see that we are not the only ones who felt the need to build a Zookeeper alternative.

Looks like folks at StreamNative did as well, with their Oxia project: https://github.com/streamnative/oxia. They were just talking about this yesterday at Confluent Current ("Introducing Oxia: A Scalable Zookeeper Alternative" was the title of their talk). https://streamnative.io/blog/introducing-oxia-scalable-metad...

Seems to be a trend :)

View on HN

looks to be a slightly different design goal on oxia wrt to replication and fault tolerance. https://github.com/streamnative/oxia/blob/main/docs/design-g...

Def.

I hadn't seen Oxia before but the idea, for their implementation, of making Zookeeper more like Bookkeeper was an interesting one.

Not right for ClickHouse needs but, IMO, a novel approach.

Meta additionally has some internal software called Zelos, which is the zookeeper api but implemented on delos[0], though you didn't hear it from me

[0] https://www.usenix.org/system/files/osdi20-balakrishnan.pdf

Is the trend mainly due to ZK being written in Java?

It's not the written in Java part, it's the running in the JVM that's the issue. Memory hungry is what I think of when I think Java apps. I'd much rather have a low memory Go or Rust service.

but if you use ZK for what it was designed for (basic cluster role/naming cordination, distributing configs to cluster nodes and similar) then this kind doesn't matter

I mean in a typical use case of it you would

- run it on long running nodes (e.g. not lambda or spot instances)

- run more or less exactly 3 nodes up to quite a cluster size, I guess some use-cases which involve a lot of serverless might need more

- configs tend to not change "that" much nor are they "that" big

what this means is

- java needing time to run-hot (JIT optimize) is not an issue for it

- GC isn't an issue

and if you look at how much memory (RAM) typical minimal nodes in the cloud have it in context of typical config sizes and cluster sizes is also not an issue

through I guess depending what you want to do there could be issues if you use it

- for squeezing through analytics or similar

- setups with very very very large constantly changing clusters, e.g. in some serverless context with ton of ad-hoc spawned up instances, maybe using WASM and certain snapshot tricks allowing insane fast startup time

- you want to bundle it directly into other applications, running on the same hardware and that applications need more memory

but all of it are cases it wasn't designed for so I wouldn't call it an "alternative" but a ZooKeeper like service for different use-cases, I guess

Long-running server processes are not only "not an issue" for the JVM, they're the main use case in which the JVM is superior to AOT compilation! Same is true for C# and the .NET CLR, by the way.

If you're running a Lambda function in which startup time is extremely important, or an embedded application where size and resources are paramount, or even just a short-lived process where you don't care either way... then AOT makes a lot of sense.

But for long-running server processes, just-in-time compilation almost always results is better performance than AOT compilation that cannot optimize at runtime based on what's actually happening.

HN should be full of people who know better, but these discussions feel like piping information into /dev/null. Web devs, students and hobbyists, and other low-information voters just have it in their heads that AOT is always a superior model, and JIT always an inferior fallback, and there's nothing you can say to break through that. There aren't enough people from the business server side world who spend enough time in online discussion forums to correct the narrative.

I agree that a good JIT and the JVM is very powerful, but it's not a silver bullet that magically works for all systems. For example, JIT-compiled code can become uncompiled and recompiled all the time in the same process, resulting in weird performance characteristics. You can definitely fine tune and code in ways that prevent this, and for many apps it won't matter, but when it does matter it's a pain in the butt.

More on this: "Virtual Machine Warmup Blows Hot and Cold" https://arxiv.org/abs/1602.00602

I think there's an important distinction to be made: Theoretically speaking, given that JIT compilers generally care much more about compilation speed than AOT ones, it is not unreasonable to assume that AOT compilers ought to produce more optimized code.

With that said, this is not the case for both JVM and .NET. Stepping away from ought to is, both have JIT compilers which produce better optimized code than their AOT counterparts, due to a variety of reasons including R&D effort done for JIT throughout their history and JIT allowing to dynamically profile and recompile the code according to its execution characteristics (.NET's Tier 1 PGO Optimized and HotSpot JVM's C2).

Theoretically speaking, JITs have access to strictly more information than AOT, so they ought to be better once you amortize out the timing issues. A really good profiling JIT will do a fast-compile pass the first time through, and then progressively re-compile with knowledge gained from profiling during runtime.

“Has more information therefor better optimized” is bordering on a lie, is the problem.

Theres truth to a point I guess, but it’s also true that lots of “more information” is useless, and sometimes harmful to optimization, and additionally that it’s really diminishing returns.

Truth is as long as you’re not over relying on RAII, JVM will virtually never outperform C++.

It's not a lie at all.

That's why there is profile guided optimizations for C/C++.

Which instruments you C/C++ code to collect that needed additional information (on the cost of performance).

Then when recompiling you can feed that collected information into your system.

The problem with profile guided optimization is that it's way more annoying to deploy as on every update you have to deploy it twice once slower then naive and then once faster. And because the slower part might very well be to slow you might want to only deploy it to some nodes of a load balancer and deploy naive compiled versions to the other.

It also means you have a similar slower => faster startup time, excepts it's of your program as whole instead of for each restart of a node.

And while optimized Java likely will never outperform optimized C++ that is Java specific furthermore if we speak about common non manual optimized code which isn't implementing some tight math/CS algorithms (i.e. very common daily code in many companies) then the difference really isn't that big, small enough to make choosing java over C++ for server stuff a "in general" the right choice. (If you are not a company which only gets the "best" programmers like google).

I mean just to put it into context there are companies which had success with stuff like running high speed trading code on the JVM and it was a success. So if you can do that probably Java doesn't have a major performance problem.

> not over relying on RAII

you mean like throwing out all the major improvements of C++ which majorly reduced the probability of non highly expert programmers introducing bugs which could be turned into RCEs?

Suggesting that you shouldn’t be allocating memory and then immediately tossing it doesn’t only suggest that you should practice RCE prone manual strategies.

RAII is dogshit because it encourages and hides the fact of what’s really happening, leading to crazy performance gotchas. Just knowing the gotchas around it is often enough for any half way competent programmer to devise better code.

RAII is the foundation of safe memory management in C++

If the "more information" is useless, it can be ignored. JIT can't possibly be a worse strategy, because a JIT always has the option of simply doing an AOT compile at program start.

That doesn't mean that in practice the JVM is faster than well-written C++; the Java language semantics almost prevent it from doing so in the general case. But in principle, if all else were equal, it should be able to be.

You've not had to diagnose ZK mysteriously hanging or pegging its CPU due to GC or mysterious memory bloat? I have

And the JVM and Java ecosystem has much better profiling tools than when Golang, e.g. has mystery GC pauses.

For a long running, memory safe, server process, there's no other ecosystem (that I know of) that is quite like the JVM.

Consider this: Kubernetes people felt etcd was a better choice, largely because of performance.

https://news.ycombinator.com/item?id=18687516

etcd is pretty good in itself, but Kubernetes has many design choices that do not translate well to other projects, or even to Kubernetes itself.

I mean java native compilation is a thing. I imagine you'd get the majority of the lift via that route instead of reinventing the wheel, but it's possible there are caveats to that route for these impls. Even java native/golang have GC heaps, but in general ZK is really low GC churn so I don't see that being the impetus to rewrite.

My guess if anything is that people always complain about ZK being annoying as a second piece of infra to distribute for simple setups, so my guess this is just a prelude to them embedding their keeper into the DB deployment itself which is the same general strategy Kafka is taking (a verrrry loong time) to rollout.

If you need a Zookeeper drop-in replacement then I suppose there’s limited options, but many of the same needs could be met by etcd.

TBH, I don't think so.

I mean, I have worked and, and been guilty of tooling driven development (RiiR anyone?) .

But, also, in a comment below Alexey shares many of the reasons other than language. I think Oxia does a good job of sharing their approach in - https://github.com/streamnative/oxia/blob/main/docs/design-g...

(Alexey's comment, FYI, https://news.ycombinator.com/item?id=37677324)

Reply on news.ycombinator.com