Basically an astute reader of Jepsen testing may deduce "I need to use a single node system" which is one option but without the availability characteristics modern users usually want
Basically an astute reader of Jepsen testing may deduce "I need to use a single node system" which is one option but without the availability characteristics modern users usually want
That's what a successful test looks like. Sure, they found correctness issues with the lock api, but they were able to lend some confidence to etcd's core api. There might be bugs, but jepsen didn't observe any.
Sure, that's not proof that systems don't have tradeoffs, but it's proof that jepsen doesn't literally always find consistency issues or other core problems. Said another way, Jepsen is testing for correctness issues in systems. They either find some or they don't. These results are interesting, even if they are not a panacea.
There are other tradeoffs, but for most systems, correctness is important enough it merits consideration on its own right.
I don't think readers of jepsen misunderstand what's being tested or what it means, nor do they misunderstand that there are other tradeoffs to consider, so I think your comment is off the mark.
By analogy, if we were reading a post about "I load-tested this bridge which claims to support 50 tons of weight, and it broke at 5 tons", no one would be saying "yeah, but there are tradeoffs for bridges. This post just makes it clear that there's tradeoffs. If you make the bridge stronger, it would be more expensive, and to imply there's not tradeoffs is misleading. An astute reader might deduce that they should never drive on bridges again".
I don't think that would be a reasonable interpretation of such a post, nor do I think the interpretation you espouse here portrays an accurate sentiment.
PS: There is also another trend, that is System claiming they fixed the issues Jepsen finds, without submitting themselves again for analysis...but I digress now...
Here's an analogy that may help me communicate how I feel since I realize my message is not landing: Let's say I'm buying a condo in the San Francisco Bay area. And let's say the building that I'm looking to buy in advertisers that it is historic but seismically retrofitted. Then say Jepsen-earthquake-test comes in and shakes the ground beneath the building and shows that the building indeed collapses with enough of an earthquake: would that or would that not be enough information for me to decide whether or not the seismically retrofitted building is good enough for my needs? There's a lot of ambiguity in answer that question.
When he demonstrated that Riak was dropping 30-70% of writes, even with the strongest consistency settings, or that Mongo had multiple scenarios of data loss, we are not talking about the subtleties of the English idiom
A Jepsen style test optimized not for bending to show where things break but instead for showing likely real world style situations with an eval of which are most likely to arise would be far more valuable for people
FWIW, I think you're raising really good questions in this thread. Qualitative safety is highly contextual, depending on fault model, significance of individual operations, concurrency, throughput, latency demands, operational characteristics, data volume, etc etc., and I try to touch on that in the "Toward a Culture of Safety" section in this report. Hopefully that resonated for you.
Rather, it would be "Builders claim their buildings are earthquake proof, and jepson was able to show a subset collapse in earthquakes. The rest may or may not be earthquake proof".
That's still very valuable. It's really valuable to know when something is wrong. It would be more valuable to know that something is definitely right (correct / "earthquake proof"), but jepsen cannot prove that.
The real world is significantly more "contrived" than anything Jepsen can come up with.
Transactions timing out, Nodes losing transactions even after they've been acknowledged etc.? None of this is contrived.
> what most users need to understand about said systems
The target audience of these reports are the system builders and the software engineers building services on top of them; not end-users consuming higher level services.
- If we're sticking with your example further up-thread, you'd buy a house that was advertised as "can withstand a 4.0 magnitude earthquake", that then failed when subjected to a 4.0 magnitude earthquake.
- 4.0 magnitude earthquakes happen all of the time [0].
More or less what I'm saying is, engineers generally don't think DBs lose data, and when they start coming up with ways that might happen (node failure, network failure, clock desync), distributed DBs assure them with algorithms and configuration knobs. Aphyr puts those assurances to the test, which is so, so valuable to us all.
It's also worth saying that this space is pretty technically complicated. All the DB engineers I know use some form of Jepsen-style testing (or Jepsen itself) because it's amazingly great.
We've been doing distributed systems for over 50 years now. There's nothing ambiguous in either the language Jepsen uses or the claims he is examining. The crypto shills chose to pretend the language is ambiguous and invent their own definitions on purpose.
Including the ridiculous "nah, everyone understands that when we speak in present tense it means we mean some unspecified point in a nebulous future".
In my opinion the Rethink DB 2.1.5 report went fairly well for that project. If the claims are aligned with the reality of the product, it's clear the Jepsen report will highlight that.