Hacker News Reader

Comment by kleinsch | Hacker News Reader

You seem to be vastly overthinking this. People have been building applications (including banking applications) using boring old relational databases for decades without losing data. Single master, replicated to another region. If relational databases were as faulty as you describe, every web app you use would be losing data all the time. Not everything has to be an overthought system design interview question.

View on HN

Single DB corruption and disk failures are more common than you think.

"Single master, replicated to another region."

Why do you recommend against using a modern distributed database?

Is it that hard to spin up something like CockroachDB? In fact, it wouldn't surprise me that Wave are running CockroachDB as their Postgres foundation under the hood.

> Why do you recommend against using a modern distributed database?

Call me maybe. This is a newer technology and the growing pains have been dramatic. I assume you've followed Jespen?

https://aphyr.com/tags/jepsen

I use MongoDB all the time, for a lot of projects. But I'm also keenly aware of how many years they have struggled to pass the Jespen tests. I can also recall several times engineers from MongoDB have said "We have finally resolved all of the issues raised by Jespen" and then another bug was discovered.

Again, I use MongoDB all the time. But I'm aware it is simpler and safer to use Postgres, for many, many use cases.

"I assume you've followed Jespen?"

Yes!

You might also want to check out autonomous deterministic testing, which is like Jepsen, but where you can explore many more state spaces by speeding up time, and where you can replay any distributed bugs instantly, just by replaying a seed.

This is a brilliant talk on these techniques:

FoundationDB — How I Learned to Stop Worrying and Trust the Database — https://www.youtube.com/watch?v=OJb8A6h9jQQ

I also work on a distributed database called TigerBeetle [1] where we use these new techniques. You can run our simulation tests yourself even. It's as simple as cloning the repo and running "scripts/vopr.sh", and let me know if you want a tour!

These testing techniques work so well that we've been running a bug bounty challenge with awards up to $8192 for anyone who can find a way to break it or crash it. It comes with the simulation testing all included so it's pretty easy to get started. [2]

[1] https://github.com/coilhq/tigerbeetle

[2] https://github.com/coilhq/viewstamped-replication-made-famou...

I'm not sure "struggled to pass" is the right characterisation. The Jepsen tests are part of the MongoDB test suite. They have found some bugs in the past and we have fixed them and incorporated into our test suite. Am I about to say Jepsen will never find another bug in MongoDB? No. But if they do we will fix those as well.

And I understand that MongoDB are also testing with Antithesis [1] and getting lots of assurance from that, right?

That gives me confidence.

[1] https://github.com/mongodb/mongo/wiki/Testing-MongoDB-with-A...

I followed MongoDB and Jespen in detail for years and I think "struggle to pass" is a correct characterization, which I believe even Kyle Kingsbury used it at one point. There was a long stretch where MongoDB treated Kingsbury as an enemy, and simply ignored him, or said he was wrong, then there were some years where they were saying "we dealt with all of the issues that Kingsbury raised" and then Kingsbury would write another blog post saying "Well, I still find these issues." Finally, MongoDB changed gears and decided to work with Kingsbury in a productive way, and then Jespen began to become part of the MongoDB test suite.

Reply on news.ycombinator.com