366 karma · joined July 10, 2009
Spanner stores its data in Colossus, so there would be some bootstrapping issues to resolve to move it to Spanner over Bigtable. (Bigtable also has the bootstrap issues but has solved them already and there are additional difficulties due to some details that I probably am not at liberty to share)
Spanner is used for metadata for many other very very large storage systems, though.
It's true that the project was initially developed at Cloudera, and employees continue to be the main driving force behind development. That said, we have committers and contributors from other companies as well. Roughly half the people who contributed a patch in the last 3 months have been non-Cloudera. Additionally we are very strict about doing all development upstream (eg with the first open source release we spent a lot of effort to open the entire development history going back to 2012, including JIRA, git, etc).
As for users, here are a couple examples off the top of my head who aren't currently paying for any support:
- Xiaomi (world's 4th largest smartphone maker) collects ~2TB/day of event data from >5million phones into a cluster which simultaneously runs analytics workloads (SQL, Spark, etc) - CERN is looking at using Kudu to store high energy physics experiment data from the ATLAS detector at the LHC. You can find some code at https://gitlab.cern.ch/zbaranow/kudu-atlas-eventindex and a poster here: https://indico.cern.ch/event/505613/contributions/2230964/at...
(of course lots more too whose names I dont have permission to mention)
Feel free to join our slack if you're interested in chatting with more - usually plenty of people online here: https://getkudu-slack.herokuapp.com
-Todd
Feel free to drop by our Slack (http://getkudu-slack.herokuapp.com ) if you have any questions.
"This application claims to the benefit of U.S. Provisional Patent Application No. 61/911,720, entitled “HYBRIDTIME and HYBRIDCLOCKS FOR CLOCK UNCERTAINTY REDUCTION IN A DISTRIBUTED COMPUTING ENVIRONMENT”, which was filed on Dec. 4, 2013, which is incorporated by reference herein in its entirety."
(which predates the creation of the cockroachdb repo and the hybrid logical clock paper).
Disclaimer: Obviously I'm not speaking for the company or making any promises here :)
-Todd
and the source here: https://github.com/apache/incubator-kudu/blob/master/src/kud...
Both are probably more readable than patent-ese :)
Some other background collector thread (which is going to actually delete removed objects) just has to wait until it sees every mutator thread cross a safepoint, at which point it knows that none of those threads could be hanging onto references that have been unlinked from the data structure.
I'd recommend reading some surveys of RCU and SMR algorithms if this stuff is interesting to you.
Basically the short answer, though, is that reads are not done by quorum, but the client can specify if they need "up to date" reads. If they do, currently, we force reads to go to the leader, but have a roadmap for how to read from other replicas (see the Spanner paper for a rough idea how that can work).
Of course this is a huge optimization for data warehousing applications, where two co-partitioned tables can be joined without any network data transfer, and in some cases could even use merge join instead of hash based strategies. But, it's the usual time/scope/quality trinity, and we'd rather not compromise the third element.
The idea is to get the analytic scan performance of Parquet while still allowing for in-place updates and row-by-row access like HBase.
HDFS (with Parquet or other formats) will still be better for unstructured or fully immutable datasets. HBase will still be better when your top priority is ingest rate, random access, and semi-structured data. Kudu should be good when you've got tabular data as described above.
So, pretty old stuff. But note that the 100G dataset fit comfortably in RAM. If you don't fit in RAM, then the latencies are obviously going to be much worse on spinning disk. Our bet is that for OLTP workloads, you're probably going to be on flash or mostly RAM-resident these days, given the dropping prices and increasing speeds of solid-state storage.
Apparently Google has this same tool for Java internally, but hasn't open sourced it due to their litigation with Oracle over Java stuff.
With a lot of effort by many folks in the community, HBase has mostly tackled the full-GC problem, but still has occasional issues with some workloads.
So, GC was definitely one factor - not having GC means we can give 99th percentile numbers in the single-digit milliseconds, which is pretty nice. Our master process actually has shown <1ms 99.99th percentile for tablet location requests on an 80 node cluster. So again, numbers that are super difficult to get on the JVM unless you take an allocation-free approach like the HFT guys do.
Another factor was ease of integration of platform-specific code for performance reasons. For example, we make use of SSE prefetch instructions to improve scan speed in our concurrent B-tree by 30% or so. The b-tree itself would be difficult to implement in Java due to lack of control over object layout, etc. While you can eventually get the same performance with enough off-heaping and sun.misc.Unsafe, my feeling is that, by the time you've gone down that road, you might as well be using C++.
I'll admit that, after many years of not writing native code, I was a bit nervous of diving back in. Segfaults are never fun. But, we soon realized that the native code tooling has improved a _ton_ in the last decade. We run all of our tests precommit using the excellent Sanitizer tools from Google (ThreadSanitizer, AddressSanitizer, LeakSanitizer) and those make it nearly trivial to diagnose a leak or crash. We also have pretty strict guidelines around use of pointers, based on the Google C++ guidelines. Many will complain that this is a neutered form of C++, and they're right. But it's also a relatively safe form of C++.
I could probably write a lengthy blog post on our experiences of C++ vs Java, but hopefully the above gives you a taste. Overall I've been happy with the decision. Slightly more time spent on crashes. Less time spent on chasing hard-to-reproduce performance or memory consumption issues. And the thread checking tools are actually far superior, so I'd say less time spent chasing races.
Throughput: 28280 ops/sec Read: 2821us avg, 467us min, 3519us 95p, 6843 99p Update: 1688us avg, 714us min, 1983us 95p, 8855us 99p
Workload D, which has some inserts (and reads recently written data):
Throughput: 36286 ops/sec Read: 1765us avg, 491us min, 2537us 95p, 4259us 99p Insert: 1614us avg, 838us min, 1595us 95p, 11575us 99p
Hope that helps. I'll try to push our latest YCSB bindings to github later this afternoon/evening if you'd like to reproduce on your own.
My guess is that the first pass will be table and column level authorization, plus of course strong authentication. Row/cell/predicate-based security could be added in a later release, but it's a feature that's less commonly required.
As for encryption at rest, I imagine that will also be fairly high priority as we move towards GA or the first few releases after GA. But again, we haven't done the scoping exercise yet, so I'm cautious to throw out dates :)
If you're interested in helping to contribute either feature, let us know! kudu-dev@googlegroups.com
I guess if your project doesn't want to include C++ (and you don't want to write a simple C wrapper), you've got a fair point. But it sounds like their implementation is significantly faster than the one in the linked post.