Hacker News Reader

Comment by redwood | Hacker News Reader

I respectfully disagree. Human language and characteristics around distributed data stores in the real world have inherent ambiguity implicit in them that Jepsen in my view pretends isn't the case.

Here's an analogy that may help me communicate how I feel since I realize my message is not landing: Let's say I'm buying a condo in the San Francisco Bay area. And let's say the building that I'm looking to buy in advertisers that it is historic but seismically retrofitted. Then say Jepsen-earthquake-test comes in and shakes the ground beneath the building and shows that the building indeed collapses with enough of an earthquake: would that or would that not be enough information for me to decide whether or not the seismically retrofitted building is good enough for my needs? There's a lot of ambiguity in answer that question.

View on HN

> I respectfully disagree. Human language and characteristics around distributed data stores in the real world have inherent ambiguity implicit in them that Jepsen in my view pretends isn't the case.

We've been doing distributed systems for over 50 years now. There's nothing ambiguous in either the language Jepsen uses or the claims he is examining. The crypto shills chose to pretend the language is ambiguous and invent their own definitions on purpose.

Including the ridiculous "nah, everyone understands that when we speak in present tense it means we mean some unspecified point in a nebulous future".

Although I agree with your point, on language and its ambiguity, I would argue that is a different claim of the one you made above, and that I replied to.

When he demonstrated that Riak was dropping 30-70% of writes, even with the strongest consistency settings, or that Mongo had multiple scenarios of data loss, we are not talking about the subtleties of the English idiom

But the contrived nature of Jepsen tests is so different than the real world. In the real world no system behaves exactly the same as it was designed to behave; the real world has cosmic rays, earthquakes and everything in between. So no statement about a software system can ever be simply accepted as a fundamental fact and so to prove that software systems break is misleading for what most users need to understand about said systems.

A Jepsen style test optimized not for bending to show where things break but instead for showing likely real world style situations with an eval of which are most likely to arise would be far more valuable for people

We do know that some databases fare much better than others and that’s useful to many. In your analogy it would be many builders claiming their buildings to be earthquake proof with only some actually being it. Thanks to Jepsen, customers know, where to buy.

I'm genuinely curious to hear which database is the winner from your perspective based on Jepsen's insight?

If you find out, let me know! Folks are always asking me "What database is best?" and I'm just shuffling my feet and going "Uhhhh, well, systems are really hard to build correctly, there's lots of tradeoffs, you probably want something with a proven replication model under the hood but so much depends on workload and operational characteristics beyond just safety... If you find a database you love please tell me about it because I like hearing people's stories..."

FWIW, I think you're raising really good questions in this thread. Qualitative safety is highly contextual, depending on fault model, significance of individual operations, concurrency, throughput, latency demands, operational characteristics, data volume, etc etc., and I try to touch on that in the "Toward a Culture of Safety" section in this report. Hopefully that resonated for you.

I think it's a refreshingly mature section, And my hats off to you for going there. Unfortunately I don't think the typical reader who I really do believe loves Jepsen outrage porn is going there

Note, this analogy is slightly off for using the phrase "some actually being it". jepsen cannot prove "earthquake proof" (correctness).

Rather, it would be "Builders claim their buildings are earthquake proof, and jepson was able to show a subset collapse in earthquakes. The rest may or may not be earthquake proof".

That's still very valuable. It's really valuable to know when something is wrong. It would be more valuable to know that something is definitely right (correct / "earthquake proof"), but jepsen cannot prove that.

> But the contrived nature of Jepsen tests is so different than the real world.

The real world is significantly more "contrived" than anything Jepsen can come up with.

Transactions timing out, Nodes losing transactions even after they've been acknowledged etc.? None of this is contrived.

Jepsen is all about verifying claims and communicating limitations. Companies hire them for this as due diligence. They fix real world bugs based on the results.

> what most users need to understand about said systems

The target audience of these reports are the system builders and the software engineers building services on top of them; not end-users consuming higher level services.

You make some reasonable points, but I would say a couple things here:

- If we're sticking with your example further up-thread, you'd buy a house that was advertised as "can withstand a 4.0 magnitude earthquake", that then failed when subjected to a 4.0 magnitude earthquake.

- 4.0 magnitude earthquakes happen all of the time [0].

More or less what I'm saying is, engineers generally don't think DBs lose data, and when they start coming up with ways that might happen (node failure, network failure, clock desync), distributed DBs assure them with algorithms and configuration knobs. Aphyr puts those assurances to the test, which is so, so valuable to us all.

It's also worth saying that this space is pretty technically complicated. All the DB engineers I know use some form of Jepsen-style testing (or Jepsen itself) because it's amazingly great.

[0]: https://research.google/pubs/pub40801/

This is such a clown take. Read any root cause analysis of your favorite cloud operator and you will find all of the scenarios that Jepsen tests for in those. And you know what's even better? They can be solved!

Reply on news.ycombinator.com