Jepsen: MongoDB 4.2.6

My only experience with MongoDB is being "the engineer called to clean the mess". I'm sure you can effectively use MongoDB in production if you're knowledgable and careful, but most people aren't and they shouldn't have to know the detailed inner working to not create a mess.

Transaction ID wraparound: https://twitter.com/bcantrill/status/1110647418008133632

Incorrect use of fsync: https://news.ycombinator.com/item?id=19119991

Also would depend on how long time you think the application will be around. You're building an MVP to evaluate something? Just hack together whatever will work (then throw away). You're maintain software for a library/archive that will most likely stick around for a long time, even if they say it's just temporary? Do decisions that will help in the future, always.

We have a roadmap we need to meet and so far we have been trying to spill money on it rather than developers (paying mongo atlas) and adding features incrementally as Mongo gets them (like transactions).

If this wasn't a startup we would probably rewrite.

  MariaDB [test]> create table test ( i int );
  Query OK, 0 rows affected (0.06 sec)
  
  MariaDB [test]> insert into test values (''), ('xxx');
  Query OK, 2 row affected, 2 warning (0.01 sec)
  
  MariaDB [test]> select * from test;
  +------+
  | i    |
  +------+
  |    0 |
  |    0 |
  +------+
  2 row in set (0.01 sec)

That's a new MariaDB 15.1 with the default settings I just installed the other day to test some WordPress stuff. I know there are warnings, and that you can configure this by adding STRICT_ALL_TABLES to SQL_MODE, but IMO it's a dangerous default.

This is also an issue with using MongoDB as a generic database: every time I've seen it used there were these kind of data integrity issues: sometimes minor, sometimes brining everything down. Jepsen reports aside, this alone should make people double-check if they really want or need MongoDB, because turns out that most of the time you don't really want this.

    withTransaction(async session => {

      await Promise.all([someOp(sesson), 

      someOtherOp(session)]);

    });

However, when the driver calls your function again it doesn't invalidate the `session` object - so previous calls to the same function can make updates to the database.

Let's say `someOp` does something that causes the transaction to retry and `someOtherOp` is doing something non-mongo-related in the meantime (like pulling a value from redis). Now `someOtherOp` reached the mongo part of its code and it is executing it happily with the same session object (so operations succeed although they really shouldn't)

The point of transactions like you said is to perform multiple operations atomically and for them to happen "exactly once or not at all". With Mongo in practice it is very easy to get "Once and some leftovers from a previous attempt".

We reported it immediately at the highest severity and we pay for the highest tier or support - we tried to collaborate as soon as possible. It sort of went "over their head".

This is a wonderful idea, and I've got no idea how to actually do it in a standardized, rigorous way. Vendor claims are often contradictory, it's hard to get a good idea of anomaly frequency, availability is... a rabbithole, and it's hard to come up with a standard taxonomy of anomalies--most of the analyses I do wind up finding something I've never really seen before, haha. With that in mind, I've wound up letting the reports speak for themselves.

Also, ideally the analyses should be rerun automatically (or possibly after being paid, but making it easy for the company to do so) every time a new major release happens rather than being done once and then being stale.

I don't know a good way to do this either. Each report is typically the product of months of experimental work; it's not like Jepsen is a pass-fail test suite that gives immediately accurate results. There is, unfortunately, a lot of subtle interpretive work that goes into figuring out if a test is doing something meaningful, and a lot of that work needs to be repeated on each test run. Think, like... staring at the logs and noticing that a certain class of exception is being caught more often than you might have expected, and realizing that a certain type of transaction now triggers a new conflict detection mechanism which causes higher probabilities of aborts; those aborts reduce the frequency with which you can observe database state, allowing a race condition to go un-noticed. That kinda thing.

If I'm lucky and the API/setup process haven't changed, I can re-run an analysis in about a week or so. If I'm unlucky, there's been drift in the OS, setup process, APIs, client libraries, error handling, etc. It's not uncommon for a repeat analysis to take months. :-(

Plus maybe a column indicating what [the company behind the database] claims?

Even common highly available configurations take the route of no consistency guarantees by doing primitive async replication and primitive failover.

The middle part of the report talks about unexpected but (almost all) documented behavior around read and write concern for transactions. I don't want to conjecture too much about motivations here, but based on my professional experience with a few dozen databases, and surveys of colleagues, I termed it "surprising". The fact that there's explicit documentation for what I'd consider Counterintuitive API Design suggests that this is something MongoDB engineers considered, and possibly debated, internally.

The final part of the report talks about what I'm pretty sure are bugs. I'm strongly suspicious of the retry mechanism: it's possible that an idempotency token doesn't exist, isn't properly used, or that MongoDB's client or server layers are improperly interpreting an indeterminate failure as a determinate one. It seems possible that all 4 phenomena we observed stem from the retry mechanism, but as discussed in the report, it's not entirely clear that's the case.

That's not to say that workarounds don't exist, just that I didn't find any in the documentation or by twiddling config flags in the ~2 weeks I was working on this report. :)

If you look at Elle's transaction generators, you can cap the size of any individual key, and use an uneven (e.g. exponential) distribution of key choices to get various frequencies. That way keys stay reasonably small (I use 1-10K writes/key), some keys are updated frequently to catch race conditions, and others last hundreds of seconds to catch long-lasting errors.

So I'm curios how would you have described the ability of finding violations with Elle using read-write registers with unique values vs the append-only lists?

RW registers are significantly weaker, though I don't know how to quantify the difference. I've still caught errors with registers, but the grounds for inferring anomalies are a.) less powerful and b.) can only be applied in certain circumstances--we talk about some of these details in the paper.

https://web.archive.org/web/20150312112556/http://blog.found...

Jepsen draws inspiration from a long line of work on property-based testing, especially Quickcheck & co. It also draws on roughly 10 years of experience building & running distributed systems in production. A lot of Jepsen I invented from whole cloth, but some of the checkers in Jepsen are derived from specific research papers, like work by Wing, Gong, and Howe on linearizability checking.

Then it's just... a lot of thinking, experimenting, and writing. Jepsen's the product of ~6 years of full-time work. Elle, the system which detected the anomalies in this report, was a research project I've been puzzling over for roughly two years.

I write the Jepsen series, and open-source all of the code for these tests, partly as a resource so that other people can learn to do this same kind of work. :-)

My bias: I like and heavily use ZooKeeper in production. HN seems not to like it as much.

Because the answer is "no" in the overwhelmingly majority of cases, specially if your product is mature.

Given that is a pretty popular part of MongoDB seems like an important thing for people to continuously fail to mention.

A detailed description of the bug can be found here: https://jira.mongodb.org/browse/SERVER-48307

This bug has been fixed and backported, and will be available to users in MongoDB 4.2.8 onwards. The MongoDB test suite has been updated to ensure that this specific phenomenon is detected in future releases. We are also planning to update the version of Jepsen we are currently running in our CI loop to include the newest test case used in the report.

Last, we’ve made some changes to how we share information discussed in the Jepsen reports on our website. You can find the updated page here (https://www.mongodb.com/jepsen).

I’d love to read a roasting like that authored by Leslie Lamport for a different perspective but aphyr’s works absolutely stand on their own.

Any ideas how to get Jepsen and TLA to work together? :)

Woah, that's wild. Are there any pre-prints/papers/talks that you can link to on this subject? I'd _love_ to read this.

> I'm also excited about my own research with Elle, but we're still working on getting that through peer review, haha. ;-)

I read over bits of Elle; the documentation in it is absolutely top-notch. You and Peter Alvaro knocked it out of the park!

Yeah, but in spite of that their performance still sucks compared to writing directly to /dev/null, and that's where Mongo steals their thunder.

You do know that PostgreSQL had issues with not fsyncing data as well ? It's technology. Bugs will be made. Design decisions will be wrong.

I think it's really disappointing and inappropriate to be labelling MongoDB engineers as unethical for simply having incorrect defaults. Which in their history they often change after they are made aware of them.

It's totally fine with me, but I just wasn't aware of it.

Of course, it doesn't work when you don't pair it with majority read/write concern. You can't expect to get a snapshot of data that wasn't yet acknowledged by majority of the cluster.

As to the quote you probably are referring to:

"Jepsen evaluated MongoDB version 4.2.6, and found that even at the strongest levels of read and write concern, it failed to preserve snapshot isolation."

I did not find any proof of this in the rest of the report. It seems this is mostly complaint of what happens when you mix different read and write concerns.

I would also suggest to think a little bit on the concept of snapshot in the context of distributed system. It is not possible to have the same kind of snapshot that you would get with a single-node application with the architecture of MongoDB. MongoDB is a distributed system where you will get different results depending on which node you are asking.

The only way you could get close to having a global snapshot is if all nodes agreed on a single truth (for example single log file, block chain, etc.) which would preclude read/write with concern level less than majority.

Imagine there was a programming language which had rather inconsistent naming, poor automated testing support, and a history of guiding its users toward security vulnerabilities. A culture would grow up around that language and the most successful members would be those who could best tolerate those properties. People generally self-select into language communities. So unless some powerful influence pushed random programmers to use the language or made it easier to add new tooling, the culture would continue to undervalue what the language originally lacked.

I suspect the same social dynamic would apply to a database.

In any case any person that has some experience with distributed systems will understand what it roughly means to get an acknowledgment from just a single node vs. waiting for the majority.

Oracle also does not use serializable as its default isolation level, yet it advertises it.

This is all part of the product functionality. Whenever you evaluate product for your project you have to understand various options, functionalities and their tradeoffs.

Defaults don't mean shit. In a complex clustered product you need to understand all important knobs to decide the correct settings and configurable guarantees are most important knobs there are.