aphyr

There's been a lot of work on both of these problems, but right now, proving concurrent algorithms correct, and proving equivalency of those algorithms to executable code, are very much open research problems.

There's another aspect to this: now that I've established a reputation and people take my work seriously, I don't have to be snarky or aggressive to get attention drawn to these issues. And the DB landscape has changed a lot in the last five years! Engineers want to provide formalized safety guarantees. Vendors are taking these failure modes seriously instead of dismissing them as irrelevant! Since I'm getting paid, I can invest more time into making these analyses real collaborations with the vendors. So I see my role now as more about helping people reach their safety goals, and a little less about shaming folks for making systems that didn't live up to their claims.

My other motivations--letting users know how to work with the databases they've chosen, and giving case studies to help other engineers test and improve their own systems, well, those are unchanged. :)

Best regards, another grumpy software & physics person. :)

Jepsen takes great pains to make no assumptions about the time it takes for transactions to execute. We test systems with millisecond-level timeouts, or multi-minute timeouts, and both work fine.

For context, Jepsen started as a series of volunteer nights-and-weekends blog posts and conference talks. About three years in I bootstrapped the business as a consultancy, with one client lined up and ~15K USD in [available] savings. Scraped through the first year by dipping into credit cards, learned a lot about pricing and pipelines, and am doing pretty well now. Having money in the bank lets me do more volunteer work, like this analysis. :)

Jepsen makes money through consulting services (usually paid analysis work), training classes, speaking engagements (I charge at for-profit orgs and speak for free at nonprofit events), and Amazon Marketplace subscriptions.

That said, it does push 50,000 reqs/sec on my local box, did just fine on an AWS m3.medium, then fell over today at 5 reqs/sec on a heroku "pro" dyno, so... this is something I'll have to address.

To speculate: if clocks do go out of bounds, I don't... think... you can violate single-key serializability, because that goes through a single Raft cluster, but you can definitely see multi-key consistency anomalies. I think you could also get stale single-key reads, because the read path IIRC relies on a time-based lease. Might be other interesting issues if Cockroach is shuffling data from one Raft ensemble to another--I think the range info itself might have a time-based lease that could go stale as well. Perhaps a Cockroach Labs engineer could comment here?

Note that the time window for these anomalies is limited to a few seconds-nodes will kill themselves when they detect clock skew.

Both depend on wall clocks.

a.) this is a beta product where the team has been focusing on completeness and correctness before performance

b.) I was testing Cockroach from August through October--Cockroach Labs has put a good deal of work into performance in the last three months, but that work hasn't been merged into mainline yet. I expect things should be a good deal faster by 1.0.

c.) Jepsen does some intentionally pathological things to databases. For instance, we choose extremely small Cockroach shard sizes to force constant rebalancing of the keyspace under inserts. Don't treat this as representative of a production workload.

That's why Cassandra offers choice at a query level.

It doesn't give you a choice: you get last-write-wins, or some limited merge functions with CQL types. You can't get generalized CRDTs in Cassandra, because they won't expose conflicts to the user layer. Cassandra gives up on a whole class of safe AP datatypes as a consequence of this restriction.

I suggest you re-read the analysis. Some databases can offer safe, generalized commutative updates; e.g. Riak. Cassandra can't: updates, in general, can be lost through reordering.

> There is no better way than last write wins if you want high availability and partition tolerance and don't want to pay the performance and availability price for a consensus algorithm like Paxos or Raft.

There is. There's a whole field of research devoted to this problem. http://hal.upmc.fr/inria-00555588/document

In 2013, I performed an unpaid analysis, in my nights and weekends, of MongoDB. I found a bug leading to the loss of acknowledged writes with majority write concern. Mongo fixed this bug within a few weeks.

In 2015, I performed a followup test as a part of my work at Stripe. I confirmed dirty reads (which were already documented, though perhaps not well-appreciated), and discovered stale reads (which ran counter to MongoDB's documentation). MongoDB wasn't enthusiastic about that report initially, but got things sorted out and started work on adding majority and linearizable read concerns. I found that writes appeared linearizable.

In spring and summer of 2016, MongoDB paid me to help expand the Jepsen tests and hook them up to their internal CI system, so they could use the Jepsen tests to help verify their ongoing work towards linearizable reads. I privately confirmed that MongoDB still failed to prevent stale reads, but the dirty-read failures I'd seen in 2015 appeared to be prevented by majority reads.

In fall 2016, MongoDB announced they were almost ready to release 3.4.0 with support for linearizable reads, and paid me to perform a full analysis so they could be more confident in the results. It passed the linearizability tests I had initially written in 2015, but I offered to expand the tests to be more aggressive. Our collaboration resulted in the present analysis, uncovering design flaws in v0 and implementation bugs in v1. MongoDB worked to develop patches prior to 3.4.0's release, and that's why it passes now. :)