CockroachDB is more scalable than Amazon Aurora for OLTP | Hacker News Reader

CockroachDB is more scalable than Amazon Aurora for OLTP

cockroachlabs.com

81 pointsmanigandham27 comments

Eh, I find these comparisons of "X on local SSDs" vs "Y on networked storage" completely meaningless.

Even worse:

"DISCLAIMER: this benchmark script was not validated and certified by the Transaction Processing Council."

Of course. Try certifying it and see if it still stands.

Look, there is like a billion db vendors trying to push their ware every year, every month. Many of them fake the benchmarks (like, not writing to the permenant storage at commit). Which is why we have standards. So let's report standard TPC-C numbers.

Author here. The point of certification is to ensure transparency. CockroachDB is open-source. Our entire benchmarking - tools, scripts, are also open source, and the results are from runs on public cloud providers. You can replay our entire benchmarking runs. We think this is more transparent than auditor certification.

Auditor certification is also time consuming and expensive. We'd have to do it with an on-premises deployment, which means that the auditor certified version wouldn't be easily reproducible the way our current benchmarks are. If we see customer demand for auditor certified results, we'd certainly be open to the idea.

I'd love to try Cockroachdb for a reasonably sized service but it's active-record adapter[0] doesn't work yet. I'm watching the repo to see what progress is made towards supporting Rails 5.

[0]https://github.com/cockroachdb/activerecord-cockroachdb-adap...

You might want to check out this PR: https://github.com/cockroachdb/activerecord-cockroachdb-adap... looks like a lot of work has been put into compatibility but it hasn't yet been merged

This is one of the dumbest blog post I've seen in years.

How can you compare two databases in this way ? Each is running on different clouds with different hardware. Aurora isn't even using Provisioned IOPS (or is it we can't even tell). And in the Large Cluster scenario they have a staggering 30x GAE instances. Where is the cost comparison here ?

Anyway this is just SEO/Content Marketing fluff. Shouldn't be on here.

It "can" be on here, IMHO just don't upvote it..(or downvote if you are really angry at some text)

I could cost more but if CRDB is able to do it without vendor lock-in I don't think it's just SEO/Content marketing fluff, the blog post invites you to try it yourself anyway so I'd do that first and then call them bs.

>> without vendor lock-in

If you buy into CRDB you are vendor locked-in (just like any other db).

What I believe the parent comment is referring to is that CRDB is opensource and can be run in any cloud or bare metal. This is what interests me. If I was strictly in GCP, Cloud Spanner is just fine.

Aurora is "wire compatible" with either MySQL or Postgres depending on how you create it. There is no "vendor lock-in".

I can not install Aurora in my servers

CockroachDB sounds amazing, and all of these blog posts are great to read, but I'm yet to see someone talk about production experiences in comment threads when the blog posts are put up.

Has anyone here actually run it in production yet?

A previous company I worked at ran it in production. No problems with cockroach per se, but the code they wrote on top of it (to manage multiple writers) was a steaming pile. I’m not convinced that, even if the shim layer was better, it would have been the optimal design decision, but cockroach itself did pretty well while handling a relatively large amount of data all things considered.

The thing I know to watch out for is the minimum latency of requests. CRDB doesn't work well for workloads which make many DB queries per 'event' (e.g. N+1 queries which can and should be avoided).

Why was it necessary to 'manage multiple writers' in that use case you encountered?

My somewhat embarassed apologies.

I have confused RocksDB with CockroachDB.

My understanding is that, at the time of development, Rocks itself did not support multiple writers. This was in 2015, or maybe 2014. At this point it looks like TransactionsDB might cover that, but I was not super familiar with that particular project, nor am I super familiar with RocksDB in general. I do know that they asserted Rocks could not handle multiple writers the way they wanted, and they needed something which handled SSTs well. Rocks was chosen and the shim layer was written on that assumption.

Thanks for clarifying. I'm also interested in RocksDB, in particular how it's being integrated into the (poorly named) WebScaleSQL MySql variant.

So 30 CockroachDB nodes running on Google Compute Engine instances compared against previously-published reports of a single Aurora RDS instance only beat the latter's reported throughput by ~10x.

Aurora already supports horizontally-scaling reads across multiple read-replicas, and multi-master clusters for horizontally-scaling writes are currently in preview.

Horizontally scaling TPC-C with sharding is nontrivial, as the TPC-C specification requires cross-shard transactions. This is the intent of the TPC-C benchmark: real world use cases seldom shard perfectly. CockroachDB scales to 30 nodes in this benchmark while hiding all this complexity under the hood.

As far as beating Aurora, CockroachDB's 3 node deployment has fewer vCPUs (48 vs 64) than an active-passive replicated Aurora RDS instance, and matches it in performance. CockroachDB also scales horizontally, as we show, from 3 nodes to 30 nodes, linearly increasing its throughput. I don't believe that 10 shards of Active-Passive pairs of Aurora would scale linearly to 10,000 TPC-C warehouses, and as far as I'm aware, Amazon has not made that claim.

Aurora's horizontal scaling doesn't depend on sharding at all- read-scaling uses read-replicas, and multi-master write-scaling uses a hierarchical conflict-resolution strategy [1].

On 'beating Aurora', my point was that comparing a 30-node CockroachDB cluster against a single-node Aurora benchmark and concluding that Cockroach is 'more scalable' is disingenuous, considering Cockroach 2.0 just reached GA, and Aurora multi-master has been available in preview since November and will likely also reach GA in coming months.

Once Aurora multi-master reaches GA, I'll look forward to a more direct comparison of an Aurora multi-master cluster against a similarly-sized CockroachDB cluster.

[1]: https://www.slideshare.net/AmazonWebServices/deep-dive-on-th...

Problem is all the benchmarks so far are on dataset sizes that are meaningless for solutions of this type. You could get almost 10X that number on a high end commodity x86 box with say PostgreSQL. It would be interesting to see something in 100TB+ size ideally closer to petabyte

this is a comparison between two things you'd use if you want to eliminate single points of failure. "you can do better if you're willing to tolerate a single point of failure!" is, uh, apples to oranges, at best? note the section comparing availability.

(i feel there is also something to be said for being able to get lots of transactions out of more, but less expensive machines. the world is full of people working under a variety of constraints.)

You do realize the number of critical workloads running on HA RDBMs clusters is N orders of magnitude higher than number of critical workloads running on CockroachDB?

oh, you meant to say that a cluster of high end commodity x86 boxes with postgresql and a bunch of HA stuff you've got to set up yourself could beat it.

spectacular, that sounds great, i'm convinced.

I meant to say that it would be interesting to see benchmarks for dataset sizes meaningful for running distributed systems.

TPC-C's intent is to make a tradeoff between two competing goals: benchmarking total data stored and processing transactions on the stored data. It's one point in the spectrum of possibilities, but one that has withstood scrutiny for many decades. I agree that it would be nice to store 100s of terabytes in a cluster, but one then also has to run transactions over that stored data.

But for perspective, TPC-C over a 100TB dataset would be 1 million warehouses, and 12 million tpmC. Only one TPC-C benchmark EVER has hit this scale, Oracle, in 2010, and had a hardware cost of 30 million dollars.

We'd like to get to that scale one day, and benchmark it, just as we have today. But it's not where the industry is at right now. CockroachDB keeps up with TPC-C 10,000: this means 800GB, replicated 3 ways across a 30 node cluster. Aurora RDS doesn't keep up with the transaction rate at a mere 800GB of data (the 10,000 warehouses number). They claim to store up to 64TB of data, but what's the point of storing all that data if you can barely run any transactions on it?

Industry for what type of databases? 100+ TB DBs are very common thing in the RDBMS world. Doesn't have to be TPC-C just some benchmark with reasonable dataset size. 2010 was 8 years ago in modern world an x86 box can have 8 CPUs with close to 200 physical cores 6TB RAM and even consumer grade SSDs can store 800 GB multiple times on a singe drive. Outside of very niche scenario why would I use distributed database if I am not sure it can handle scale that plain old RDBMs can handle? The interesting use case is dataset size that a plain RDBMs would not be able to handle.

But more scalable than Spanner?

Reply on news.ycombinator.com