Call Me Maybe: MongoDB Stale Reads
aphyr.com
aphyr.com
https://github.com/scalien/scaliendb
Outcome: you've never heard of ScalienDB; MongoDB brilliantly won by winning the hearts and minds of hackers and coders who don't care about such issues, but were able to get started quickly with Mongo (and got cool free cups at meetups). It turns out that's most engineers out there, definitely the initial critical mass to target for a database startup like Mongo.
Btw. the story behind Oracle is similar: early versions were basically write-only; read Ellison's book 'Softwar'. Of course there are other ways to get started: for example DBs coming out of academic research like Vertica seem to avoid this problem; in that case initial funding is basically provided by the gov't and when they create the company to commercialize they're already shooting for Enterprise contracts, skipping the opensource/community building phase of Mongo.
MongoDB targets a part of the market that is well served. There are loads of other options that work well like PostgreSQL. If you were targeting a truly distributed transactional database, you might have been trying to build something that has that order-of-magnitude-better characteristics, but MongoDB doesn't seem like that.
Vertica, on the other hand, really targeted an area that wasn't well served and a decade after C-Store this area still isn't well served by anyone other than Vertica. ParAccel/Redshift isn't really comparable, though many go with it because Amazon will manage it for them.
For MongoDB, the question is "how can we win over developers' hearts and minds?" Why use Mongo? Well, you like the query DSL and get annoyed by joins, and they have a great website and documentation, and we all know that joins are slow so without them everything must scale nicely. But realistically, there are a bunch of other options that could be substituted for it.
And you're right that Vertica skipped the community aspect. The reason why is that something like Vertica isn't interesting if you don't have a lot of data. MongoDB targeted people who might have GBs of data that they wanted to store. There's a lot of people and community relations is going to matter, especially because there are lots of battle-tested alternatives. Since Vertica is most interesting when you have more data than can fit on a single machine's hard drives and might take hours to run analytics queries against, they're going to be targeting a much different market.
Vertica also had the advantage of being associated with the biggest names in databases. Stonebraker has a Turing award. Even beyond that, the names you'd think of when you think databases were involved in C-Store and Vertica.
If you're really creating something that has order-of-magnitude-better characteristics, you probably shouldn't target people that have 10GB of data to store. Someone once said something to the effect of: once there were more than 200 websites on the internet, it made more sense to target web frameworks at the websites that weren't the top 100 than solving for the problems of the top 100 sites. Most sites aren't going to need a massively distributed database. They might have dreams that the same codebase will scale from 100,000 visits/day to Facebook-scale, but they're just dreams. The fact is that for 10GB worth of data, your database probably wouldn't be better simply because that's already well-handled. In fact, distributed consensus on such small data would probably make your database worse. You really need to target people that you're solving a problem for.
Vertica just isn't interesting technology for the vast majority of users and so they don't sell to the vast majority of users. MongoDB is interesting for those who want an alternative DSL and such.
As a result of this experience I almost exclusively use PostgreSQL, and I've never EVER been burned by taking this approach.
Sometimes I do use another DB but there has to be a seriously good reason for it.
Also about MySQL, many people forget it isn't ACID compliant either and just look at benchmarks or use MySQL "because Facebook uses it". I am sure that if any startup would use PostgreSQL it would avoid many problems on the road.
But MySQL not being ACID compliant is flat out wrong (assuming that you're using InnoDB)
Somewhat later, the open source repo scaliendb got pulled only to resurface years later. What happened?
Anyway, the reason it was unstable was because we were early in the lifecycle, and as I described in the post, we couldn't get traction, so we also couldn't get investment, so it was just the two co-founders working on it. That's not enough, you need more manpower to do extensive testing of a distributed database which has its own storage engine, esp. when all of it is written in C++ for high-performance. Eg. we only had money to buy 3 servers, we used virtualization to test, but a virtualizated env. is _not_ a proper way to test a database.
https://github.com/elastic/elasticsearch/issues/2488#issueco...
As an example, HashiCorp included Jepsen test results in their Consul documentation: https://consul.io/docs/internals/jepsen.html
He even called them out for modifying timeout value to pass the tests[1].
[1] https://aphyr.com/posts/316-call-me-maybe-etcd-and-consul
This doesn't just apply to Jepsen but to all software tests. You only ever test a finite set of scenarios, so you can't really ever guarantee your software is 'safe'/bug-free - only that it does not fail in common/expected scenarios.
Kyle often mentions the tool TLA+ which enables complete and total formal analysis of certain systems. This can be a test which is complete and therefore a positive proof of correctness.
It's also not difficult to test smaller components of your system if you note that the state space here is small enough to be exhausted. This is a very important thing to look in to in serious, important components of a software system.
Finally, though it's futuretech today still, dependent types offer a general system for embedding proofs of correctness directly into code thus elevating positive proof to a computational artifact just like any other.
I once worked with a relatively well known, prove everything developer that you might know by name. He's written books and everything. He built an algorithm to make a distributed system of equal peers figure out when it needed to start more nodes, or could shut some of them down. He wrote a proof, in code. He wrote a paper. When in production, the system would not work as advertised, and he blamed it on other pieces, because the algorithm was proven correct! So the problem stayed there for months.
After he left the company, I decided to figure the problem out, so I read through the proofs, the paper, and the code: All the single letter variables you could possibly want. I figured out that yes, the algorithm was flawless, as long as every operation in the system was atomic and instantaneous. Instead of the proof, I built a small simulator that didn't have such flawed assumptions, and got the exact same behavior as the production system. So the proof was perfect, as long as we made assumptions that are impossible in our universe. And the entire algorithm was less than 200 lines of code.
So whenever we have a reality that is difficult to model (and let me tell you, distributed systems fit the bill), dependent types will not save you, haskell or no haskell. Proofs will always be limited by your assumptions.
So while the tools you mention are nice. They hit the same limits as everything else we build. Whether to write a proof in idris, use generative testing, or just some example testing, is really all a tradeoff, but you will never escape from bad specification, as all specifications are bad.
But that said, the style of analysis is totally different in each case. It's not true that you can never prove correctness. It's just a certain option with certain tradeoffs.
And there are success stories! Tons of older ones back when proof was a major component of compute programming. More modern ones like validated C kernels and compilers.
* You should never trust, and always verify, the claims made by database manufacturers.
* Especially when those claims relate to data integrity.
* Super-especially when every safety level provided by the manufacturer that includes the word "SAFE" is actually unsafe.
http://blog.foundationdb.com/call-me-maybe-foundationdb-vs-j...
Sadly, fdb has been bought by Apple[1], and you can't download it anymore. I sincerely hope foundationdb gets opensourced or something.
[1] http://techcrunch.com/2015/03/24/apple-acquires-durable-data...
Jepsen tests cannot be used to prove the system is safe, but to prove it isn't.
It looks like very often he looks into source code in order to figure out how the system operates, that way he can find weaknesses and write test to his testing framework to demonstrate the issue.
I wouldn't trust any company that uses Jepsen to show that their product is safe.
Deleted comment
[1] https://github.com/rethinkdb/rethinkdb/issues/1493#issuecomm...
> Even though the Postgres server is always consistent, the distributed system composed of the server and client together may not be consistent. It’s possible for the client and server to disagree about whether or not a transaction took place.
If you want to avoid that kind of problem, use 2PC.
Do you (or the author) see this as a bug, or just something that might surprise people who haven't thought through the guarantees?
Quoting the article:
> The 2PC protocol says that we must wait for the acknowledgement message to arrive in order to decide the outcome. If it doesn’t arrive, 2PC deadlocks. It’s not a partition-tolerant protocol. Waiting forever isn’t realistic for real systems, so at some point the client will time out and declare an error occurred. The commit protocol is now in an indeterminate state.
It would be foolish for a client to issue a COMMIT and then assume the transaction aborted because of a connection drop. The client should wait until the connection can be reestablished and determine the real transaction state before making a decision based on it.
It's the same issue as with a power failure during fsync. The durability of that transaction is indeterminate, but it doesn't matter because the system is down. Before the system comes back, it will go through recovery, and either find the commit record or not, thus getting back to a determinate state.
"During the network partition, no requests are successful" is not the best result for a CP system, IMO."
HBase should provide partial availability in the face partitions.
Also, if you follow the rest of the twitter conversation you may realize, as they did, that only requests to the minority partition are unsuccessful - which is exactly what you want from CP.
If you look at his articles: Redis, PostgreSQL, Cassandra, ElasticSearch etc all had data consistency errors. And none of those have vendors making any claims.
It's pretty sobering to say the least.
https://aphyr.com/posts/282-call-me-maybe-postgres
There were no acknowledged writes lost. The only unacked-but-successful writes resulted from a connection while a commit ack was in-flight. That doesn't qualify as a data-consistency error, it means the client has to check if the data is present after reconnecting.
But in no cases would the client reconnect to find that there were acknowledged-as-committed records that were missing or stale. In no cases would the client find that responded-as-rolled-back data was actually committed. This is very, very different than what is seen with MongoDB.
Furthermore, the postgres experiment only checked that no writes were lost. As Aphyr acknowledges, MongoDB did not lose any writes with "majority" write concern. The postgres experiment did not include the verification of a linearizable history of reads, which is what the bulk of the OP is about.
I'd like to see a similar experiment run against a replicated postgres configuration with auto-failover.
My broader point was that you need to assume the worst from your database in the application layer. Which I think you missed.
That article has various issues, for example, calling Postgres commit protocol as a special case of two phase commit is not really correct. Postgres has 2pc: http://www.postgresql.org/docs/9.2/static/sql-prepare-transa... but that was not tested.
The described behavior is "expected" and "understood". Saying that "you should assume worst from your database" is not something I would ever use for describing DB with ACID semantics.
And ACID does NOT guarentee that you will not lose data. It is a theory not an implementation. I have lost data with both Oracle and Teradata due to bugs.
Aphyr article about Postgres could be renamed to call-me-maybe-acid-db-over-the-network and could remain the same.
I understand you lost data with Oracle and Teradata.
1. Most big corporate vendors do not have systems which are very well designed. (1) The sale is made at a business level. (2) Most Oracle customers are not very tech companies, and have mixed quality employees. As a result, there is little pressure on building a robust, correct product, rather than one which meets a feature checklist. In addition, Oracle doesn't really recruit smart people (I know people who work there). It's just not very robust compared to something like PostgreSQL, which was written by Stonebraker, a legendary computer science professor and entrepreneur.
2. Still, more likely, the reason you lost data is because you didn't know what you were doing. Words like Eventual Consistency, ACID, Two-General's Problem, etc. are not just abstract. They have strict, formal meanings, and you need to understand what they do and do not guarantee. Otherwise, you will lose data again.
Missed this reply and thought it was funny. I work for one of the world's largest retailers and we are one of both Oracle's and Teradata's most loved customers. We have 4 Teradata DBAs provided BY Teradata amongst a team of 20 SQL Developers. We aren't messing around.
What YOU don't seem to understand is that bugs in your database can cause data loss. ACID or Strong Consistency will not save you.
Regardless, your comment was about PostgreSQL, not Oracle. Oracle is a giant piece of software written by a corporation with over 100,000 employees. Something like that is bound to have bugs, and it has bugs indeed. Data corruption with Oracle certainly happens. PostgreSQL is written by a small, ultra-elite team. It's a much smaller codebase, so an expert developer can understand how the whole system works. There's a big difference in robustness between the two.
Of course database bugs can cause corruption. I've certainly had MonogDB eat my data. But the level of robustness of different databases is very different. There are many databases which are essentially bug-free. If you're losing data with PostgreSQL, odds are you're the one losing the data, not PostgreSQL.
https://aphyr.com/posts/285-call-me-maybe-riak
Now I hear they support consistency as well.
[0] - http://www.kdnuggets.com/2015/03/interview-dave-mccrory-bash...
Here's what I mean. Think of all the nice things you expect to come out of Riak's theoretical basis -- bulletproof distributed writes, for example.
Well, the default last-write-wins writes aren't bulletproof. They clearly fail Jepsen [1]. And if you turned off last-write-wins, then you'd have to handle siblings -- and how you were supposed to handle them without introducing inconsistency was quite unspecified. Riak clients gave you no help there.
Or, as the Jepsen article says, you could use CRDTs. Before 2.0, Riak had exactly one CRDT, the counter.
Something that might be appealing to some is built-in MapReduce. On the forums, people would warn you not to actually use it unless you didn't actually want availability after all. I don't know if they ever sorted that out.
Another supposedly nice thing was Riak Search. A distributed DB with full-text search out of the box -- that sounds great, right? But there were two things called Riak Search, and the first one just plain didn't work. They deprecated it before it had a replacement, but the replacement was supposed to come in 2.0.
So, Osborne effect. When people gradually discovered that Riak 1.x was bad, and the response was "but Riak 2.0 will be great!", that's a great reason not to use 1.x. 2.0 took a very, very long time, enough time for customers and potential customers, including us, to find other solutions.
The original Riak Search in vs 1.x was replaced with integrated Solr in Riak 2.x, and it works.
I'd love to hear more about your use case. Contact info in profile. I'm this username in all the usual suspects.
Disclaimer: I work at Basho.
Not that it matters to me anymore, but it does sound like having all data in CRDTs is the only way to pass Jepsen. (Or to have your data be immutable, in which case Jepsen doesn't apply.)
Basho needs to sell damn fast or just call it a day and open source the enterprise version.
Well in that case have you heard about LeoFS? Wonder if it overlaps with any of the Riak CS features for you.
Contact info in profile, this username at all the usual suspects.
Disclaimer: I work for Basho.
If you wanna talk about it I'm this username at all the usual online places.
Disclaimer: I work for Basho
I'm this username in the usual places online if you wanna talk about it.
Disclaimer: I work for Basho
Given this, it is so tragic to see how dismissive they have been in regards to the consistency issues that have plagued the db since the early days. Whether it was the stupidity of bad defaults in drivers to not confirm writes, or easily corruptible data in the 1.6 days, or now with not seriously looking at the results of jepsen, the mongodb organization has never taken the issues head on. It would be so refreshing to see more transparency and admitting to the faults rather than wiggling around them until eventually pushing a fix buried in patch notes.
I often feel like a mongodb apologist when I admit that I don't mind using mongo for small (and not important) projects and while the mongodb hate can be a bit extreme at times, the companies treatment of these sorts of issues may justify some of it.
For instance, compare MongoDB's response to elastic's response:
Initial response to "Call me maybe: Elaticsearch":
https://www.elastic.co/blog/resiliency-elasticsearch/
Their ongoing status on resiliency:
http://www.elastic.co/guide/en/elasticsearch/resiliency/curr...
That is how you respond to a negative jespen test. It's particularly illuminating since elasticsearch doesn't actually bill itself as primary storage and they take resiliency seriously as opposed to MongoDB who do consider themselves primary storage and they do not.
Can also link simply to the HN discussion from back then too: https://news.ycombinator.com/item?id=1496035
> Full disclosure: I work for 10gen.
> We did this to make MongoDB look good in stupid benchmarks.
That said, I think that the 'how fast can you write to a socket?' default setting was probably intentionally put in place to make benchmarks look good.
https://jira.mongodb.org/browse/SERVER-17975?focusedCommentI...
When I run `lein install` and then `lein test`, I get:
╰─▶ ψ lein test
Exception in thread "main" java.io.FileNotFoundException:
Could not locate jepsen/db__init.class or jepsen/db.clj on classpath: ,
compiling:(mongodb/core.clj:1:1)
at clojure.lang.Compiler.load(Compiler.java:7142)
at clojure.lang.RT.loadResourceScript(RT.java:370)
at clojure.lang.RT.loadResourceScript(RT.java:361)FWIW, this was filed the other day: https://github.com/aphyr/jepsen/issues/52
https://github.com/fj/dotfiles/blob/master/home/.config/shel...
Edit: On further reading, I think you seem to be thinking that the paste of my console is an image. It is just text. To make fixed-width text on HN, indent each line by four spaces, like this:
This text has four spaces at the beginning of its line."MongoDB, even at the strongest consistency levels, allows reads to see old values of documents or even values that never should have been written."
1. http://docs.mongodb.org/manual/reference/glossary/#term-stri...
Great for sub-linear time algorithms! At that point, all my algorithms ran at less than O(n) on the size of the data I had written in.
From a business perspective, Occasional Consistency is also a very nice property if you are storing audit data for certain types of organizations. It gives complete plausible deniability about rule compliance.
I guess you could also call it Quantum Consistency.
I can't even google this term without it auto correcting to Eventually Consistency.
The wiki article, https://en.wikipedia.org/wiki/Consistency_model, doesn't even have such a term.
Doing a hard search term on google reveals nothing.
Should the general advice just be "store in MongoDB everything that doesn't require consistency and use Postgresql for everything else"?
Then, try to understand the mess Edgar Codd tried to fix in the '60s and '70s.
The actual impact of an inconsistent state is very hard to judge. It could be as minimal as having two different users just see something weird on their screen for a moment and then it goes away. It could even be totally avoided if your application handles inconsistent data well.
At the same time, it could also cause complete and nearly untraceable complete corruption of all data in your system. Who knows?
It'd be a bit like building a bridge using metal with a known defect. It'll probably work fine for a long time and depending on how and where that metal was used you might be alright.
Or you might have a complete structural integrity failure at any moment once stress starts ramping up and you'll just have to blame it wholesale on using bad materials.
How these problems will actually affect you depend on your applications. It could for example allow 2 users to be created with the same email address
Users could end up seeing private information in each others' accounts, for example.
Does anyone know of any open-source project that would aim at doing the same, so that future NoSQL DB can finally be built on strong foundations ?
Do you think it's still relevant to be bringing this up ?
But somehow, people simply started using them everywhere? Assuming that these DBs are just like any other? And now, we're all bashing on MongoDB because it is - not consistent? What happened here? :)
NB that I do not wish to attack the OP - if MongoDB now claims to be consistent in any way, that deserves scrutiny. And these analyses are always a really interesting read. But the general tone in the developer community about MongoDB seems a bit irrational.
There are ways to do "eventual consistency" responsibly. Mind you, it's obnoxiously tricky to do it right, even when someone has provided an underlying implementation that works exactly as promised. But if you design your data access patterns in the right way, the system can provide guarantees so that even if it doesn't have all your data at the moment, you can still ask questions about the the state of the data that is available, and get meaningful responses back that conform to a certain set of guarantees.
What happened here -- why we make fun of MongoDB -- is that it doesn't provide many promises like that, and even when it does, its implementation does a very, very bad job of delivering them ... and it doesn't even do a good job of delivering scalability. (It's basically a mmap()'d series of b-trees of BSON documents, so as soon as you run out of RAM, you're at risk of having the kernel swap out all your indicies instead of your data, whereupon performance craters. Oh, and the much-mocked global write lock has finally been replaced with a per-database write-lock in recent versions.)
In short, you sacrifice everything and gain... a modestly convenient API for document-storage, maybe.
It has been sold that way. People believe with write-concern majority, they can use mongo replicas for primary data stores and not lose anything. Embed your relational data and use it for everything.
We've developed a type of distributed database at my company, and I think it's pretty solid, but I need a broader familiarity with the available theory.
Academic papers will talk about "minimal" problems like consensus, or desirable properties like sequential consistency, and expect you to already know why those concepts are important. If your experience is mostly hands-on, it may not be obvious how it all applies to real-world systems.
Say you have a complex distributed database. Forget all the bells and whistles: can it solve the problem of allowing a set of processes to reliably agree on a single Boolean value? If so, then you're trying to provide the same consistency guarantees as Paxos/Raft. So if your architecture is substantially simpler than Raft, then either you've come up with something really ingenious, or you've missed some edge cases.
I would be very skeptical of any database that was written by someone who didn't have a sound foundational understanding of distributed systems theory. This is quite simply one of the places in software engineering where subtle differences in promise, protocol and expectation can make a huge difference.
Issues arise when network partitions interrupt communication between nodes; even if you require all nodes to send acks when writing, how do you deal with those acks not being received?
Or would publishing the results of doing so bring down an army of Larry's lawyering henchmen? "You violated the EULA, now you must pay! You will only wish you were dead when we are done with you, bwahahhahahaha!"
Now off to deface a piece of corporate art... :-)
Nice writeup either way though. Would like to see a similar article for Couch* and MySQL.