Kudu – Fast Analytics on Fast Data

The insert latency seems to be related to the random read latency (seems that the unique key constraint has that effect). Do you have some data on the insert latency distribution?

Thanks, Cosmin

With a lot of effort by many folks in the community, HBase has mostly tackled the full-GC problem, but still has occasional issues with some workloads.

So, GC was definitely one factor - not having GC means we can give 99th percentile numbers in the single-digit milliseconds, which is pretty nice. Our master process actually has shown <1ms 99.99th percentile for tablet location requests on an 80 node cluster. So again, numbers that are super difficult to get on the JVM unless you take an allocation-free approach like the HFT guys do.

Another factor was ease of integration of platform-specific code for performance reasons. For example, we make use of SSE prefetch instructions to improve scan speed in our concurrent B-tree by 30% or so. The b-tree itself would be difficult to implement in Java due to lack of control over object layout, etc. While you can eventually get the same performance with enough off-heaping and sun.misc.Unsafe, my feeling is that, by the time you've gone down that road, you might as well be using C++.

I'll admit that, after many years of not writing native code, I was a bit nervous of diving back in. Segfaults are never fun. But, we soon realized that the native code tooling has improved a _ton_ in the last decade. We run all of our tests precommit using the excellent Sanitizer tools from Google (ThreadSanitizer, AddressSanitizer, LeakSanitizer) and those make it nearly trivial to diagnose a leak or crash. We also have pretty strict guidelines around use of pointers, based on the Google C++ guidelines. Many will complain that this is a neutered form of C++, and they're right. But it's also a relatively safe form of C++.

I could probably write a lengthy blog post on our experiences of C++ vs Java, but hopefully the above gives you a taste. Overall I've been happy with the decision. Slightly more time spent on crashes. Less time spent on chasing hard-to-reproduce performance or memory consumption issues. And the thread checking tools are actually far superior, so I'd say less time spent chasing races.

The idea is to get the analytic scan performance of Parquet while still allowing for in-place updates and row-by-row access like HBase.

HDFS (with Parquet or other formats) will still be better for unstructured or fully immutable datasets. HBase will still be better when your top priority is ingest rate, random access, and semi-structured data. Kudu should be good when you've got tabular data as described above.