Hacker News Reader

Comment by redwood | Hacker News Reader

But the contrived nature of Jepsen tests is so different than the real world. In the real world no system behaves exactly the same as it was designed to behave; the real world has cosmic rays, earthquakes and everything in between. So no statement about a software system can ever be simply accepted as a fundamental fact and so to prove that software systems break is misleading for what most users need to understand about said systems.

A Jepsen style test optimized not for bending to show where things break but instead for showing likely real world style situations with an eval of which are most likely to arise would be far more valuable for people

View on HN

You make some reasonable points, but I would say a couple things here:

- If we're sticking with your example further up-thread, you'd buy a house that was advertised as "can withstand a 4.0 magnitude earthquake", that then failed when subjected to a 4.0 magnitude earthquake.

- 4.0 magnitude earthquakes happen all of the time [0].

More or less what I'm saying is, engineers generally don't think DBs lose data, and when they start coming up with ways that might happen (node failure, network failure, clock desync), distributed DBs assure them with algorithms and configuration knobs. Aphyr puts those assurances to the test, which is so, so valuable to us all.

It's also worth saying that this space is pretty technically complicated. All the DB engineers I know use some form of Jepsen-style testing (or Jepsen itself) because it's amazingly great.

[0]: https://research.google/pubs/pub40801/

This is such a clown take. Read any root cause analysis of your favorite cloud operator and you will find all of the scenarios that Jepsen tests for in those. And you know what's even better? They can be solved!

Jepsen is all about verifying claims and communicating limitations. Companies hire them for this as due diligence. They fix real world bugs based on the results.

> what most users need to understand about said systems

The target audience of these reports are the system builders and the software engineers building services on top of them; not end-users consuming higher level services.

> But the contrived nature of Jepsen tests is so different than the real world.

The real world is significantly more "contrived" than anything Jepsen can come up with.

Transactions timing out, Nodes losing transactions even after they've been acknowledged etc.? None of this is contrived.

We do know that some databases fare much better than others and that’s useful to many. In your analogy it would be many builders claiming their buildings to be earthquake proof with only some actually being it. Thanks to Jepsen, customers know, where to buy.

I'm genuinely curious to hear which database is the winner from your perspective based on Jepsen's insight?

If you find out, let me know! Folks are always asking me "What database is best?" and I'm just shuffling my feet and going "Uhhhh, well, systems are really hard to build correctly, there's lots of tradeoffs, you probably want something with a proven replication model under the hood but so much depends on workload and operational characteristics beyond just safety... If you find a database you love please tell me about it because I like hearing people's stories..."

FWIW, I think you're raising really good questions in this thread. Qualitative safety is highly contextual, depending on fault model, significance of individual operations, concurrency, throughput, latency demands, operational characteristics, data volume, etc etc., and I try to touch on that in the "Toward a Culture of Safety" section in this report. Hopefully that resonated for you.

I think it's a refreshingly mature section, And my hats off to you for going there. Unfortunately I don't think the typical reader who I really do believe loves Jepsen outrage porn is going there

Note, this analogy is slightly off for using the phrase "some actually being it". jepsen cannot prove "earthquake proof" (correctness).

Rather, it would be "Builders claim their buildings are earthquake proof, and jepson was able to show a subset collapse in earthquakes. The rest may or may not be earthquake proof".

That's still very valuable. It's really valuable to know when something is wrong. It would be more valuable to know that something is definitely right (correct / "earthquake proof"), but jepsen cannot prove that.

Reply on news.ycombinator.com