Hacker News Reader

Comment by kenwalger | Hacker News Reader

MongoDB 3.4 passes the rigorous and tough Jepsen test. Jepsen designs tests to make databases fail in terms of data consistency, correctness, and safety... MongoDB 3.4 passed through their newest tests.

I think that this really shows how mature of a Database MongoDB is.

View on HN

While you are correct that 3.4 now passes, I feel like your interpretation here is a bit optimistic. It's not like I tested Mongo and it passed out of the box--it failed Jepsen tests, and not just with a read anomaly--it lost majority-acknowledged inserts. The v0 protocol still fails--it's fundamentally broken. The v1 bugs are fixed now, but that's a consequence of our collaboration.

I'd say the marker of maturity here is that MongoDB has put significant time and effort into correctness: they take clock skew and network partitions as serious failure modes, they've redesigned their replication protocol, added options for stronger reads, and invested in their own correctness test suite, and Jepsen tests, as a part of their CI process.

I'd like to use the occasion to thank you for the service you provide to the community. The tests themselves, the collaboration with vendors, the competition it fosters and the in-depth technical write-ups I think push the whole ecosystem forward, which everyone eventually benefits from. Much obliged.

Thanks. :)

I agree -- I think the two most important people currently in software are aphyr for database testing, and lcamtuf for fuzzing (AFL). Both are pushing important fundamental software to pass the kind of tests it's embarrassing we haven't been doing for the past 20 years. I mean that in the most positive way, it's hard to get people to do things they've been neglecting for so long!

Note it’s not really “The Jepsen Test” as the tests are all a bit custom and different. Sometimes they’re looking to validate different claims.

For example, Kyle integrated some serializability tests into the VoltDB Jepsen testing that wouldn’t apply to Mongo.

> I think that this really shows how mature of a Database MongoDB is.

Or that it took this long for them to pass basic proficiency tests. How do other database systems fare with these tests?

You can check out the other databases he has tested here: http://jepsen.io/analyses

That being said, none of the big players in sql are there, so you can't size it up against postgres or mysql.

isn't that because they are not distributed systems?

Galera was tested - enabling MySQL to behave like a distributed database

Postgres has been analyzed, I'm not sure why it's not on that list.

https://aphyr.com/posts/282-call-me-maybe-postgres

If you look at past Jepsen test mostly they fail, and sometimes they fail spectacularly. This is one of those things where if you don't have an automated test suite to continually verify correctness you will end up with bugs.

> In addition, we show that the new v1 replication protocol has multiple bugs, allowing data loss in all versions up to MongoDB 3.2.11 and 3.4.0-rc4. While the v0 protocol remains broken, patches for v1 are available in MongoDB 3.2.12 and 3.4.0, and now pass the expanded Jepsen test suite.

Wait, it is me not understanding what the abstract is trying to say?

I can only think that the article was a bit late and the new MongoDB version has the last word. Beyond this, the article is extremely valuable and it is mandatory to upgrade your MongoDB.

This analysis was a collaboration with MongoDB over the past three months. MongoDB was able to fix these issues in 3.4.0 because I found them in rc3 and rc4.

Sorry, my fault.

It's saying it was fixed in the final 3.4.0 release (or, maybe, that there's at least a patch available) for the v1 protocol, but that the v0 protocol is unfixable - so, 3.4.0 + V1 protocol and it's solid.

Giving aphyr some money to do this makes me think much more of Mongo now; I wouldn't previously have considered it for serious use but this is an excellent demonstration of intent.

It's great that they're serious and committing resources to passing these tests. They just did, though, after nearly a decade of development. Which is terrific but it doesn't really show the 'how mature of a database MongoDB is'.

It's one test, paid for by MongoDB. It's good that they work with the dude. But there's only one test in town.

I want to see more than just one test from more than one source and seeing their sketchy performance in the past only time will tell if this new version will be any better.

> I think that this really shows how mature of a Database MongoDB is.

Yeah no. It's a test. We have to wait and see in productions and see the data to prove that. It's also the first version to past this test. It means something but not a whole lot without real production data and other people attesting to it.

Reply on news.ycombinator.com