As engineers we focus too much on the implementation details and not the benefits to the user.
How about:
- ZooKeeper alternative with lower latency
- ZooKeeper alternative with lower memory use
- ZooKeeper alternative with predictable overheads
(I don't know if these are true, just suggestions)
1. Snapshots and logs take much less amount of space on disk due to better compression.
2. No limit on the default packet and node data size (it is 1 MB in ZooKeeper)
3. No zxid overflow issue (it forces restart for every 2 bn transactions in ZooKeeper)
4. Faster recovery after network partitions due to the use of a different distributed consensus protocol.
5. It uses less amount of memory for the same volume of data.
6. It is easier to setup, as it does not require specifying the JVM heap size or a custom gc implementation.
7. A larger coverage by Jepsen tests. (This could be hard to believe, but true - ZooKeeper is tested by Jepsen, but Keeper takes the existing tests and adds more).
8. The possibility to store snapshots and previous logs on S3.
C++ isn't a key detail, just a consequence of the fact that the main ClickHouse code base is written in C++.
If you need a distributed consensus system but not necessarily compatible with ZooKeeper, there are plenty of options: Etcd, Consul, FoundationDB...
I like your suggestions!
Some of the benefits we summarized in this summary page https://clickhouse.com/clickhouse/keeper include ease of setup and operation, no overflow issues, better compression, faster recovery, (dramatically) less memory used, etc..
There was actually a reason why C++ was important for us at ClickHouse, and it's because C++ is our main code base and managing a Java project as part of it was not natural, but you right - for standalone use of this alternative, that doesn't matter.
A more revealing question is, "How are you dealing with memory safety in this implementation?" There are ways to improve memory safety in C++ through tooling and idiomatic style. Are these things being used?
There are Keeper only tests, but we run ClickHouse with Keeper for all of our server tests.
For each test we try to use all useful tools for verifying safety and correctness like sanitizers.
E.g. an interesting tool we introduced in our codebase for thread safety https://clang.llvm.org/docs/ThreadSafetyAnalysis.html#
We found some issues using sanitizers in our codebase and NuRaft library itself which were instantly fixed.
And let's not forget about Jepsen which showed some really tricky bugs but were more related to the correctness.
I would suggest looking into CBMC and similar tools as well. Model checking is incredibly useful.
The real key for using it, in my opinion, is to isolate individual classes and functions. Avoid instrumenting code with recursion and loops, and focus on defining and verifying function contracts, class invariants, and resource / memory lifetimes.
It will require a significant amount of work to mock up standard library and third party library APIs, but the real beauty of CBMC is that once you define the interface contracts for these APIs and libraries, you can verify every use of them.
I used CBMC previously to verify proper usage rules with C / JNI integration. JNI can be one complicated beast, and CBMC handily managed rule checks for its use.
I'm an extremely careful developer who unit tests everything and strives for 99% coverage. CBMC was still able to detect a memory overwrite flaw in a networking library I wrote that was based on undefined behavior due to integer promotion and offset math. This passed the various sanitizers and unit tests I had in place, but CBMC was able to reduce it to an actual crash condition that was potentially exploitable.
I don't think I can over-emphasize the usefulness of this tool.
We'll look in to them and adding to some of our social promotion over the coming weeks. Will try to find a way to give you credit.
Also, we apply many different faults in our Jepsen tests which are run 3 times a day and we never had a problem with leader election. I know this doesn't confirm that there is no bug in it but it's pretty reassuring I would say.
But the latest bugs found by ClickHouse continuous integration system in the related library were fixed about a year ago:
https://github.com/eBay/NuRaft/pull/373 https://github.com/eBay/NuRaft/pull/392
Several distributions to chose from.
You can use the OpenJDK distro shipped in your Linux distro (RedHat, Debian, etc.), you can use Microsoft's OpenJDK distro[1], you can use the Eclipse OpenJDK distro, you can use Amazon's OpenJDK distro [3] and there are a whole bunch more.
[1] https://www.microsoft.com/openjdk [2] https://adoptium.net/ [3] https://aws.amazon.com/corretto/