MongoDB and Riak, In Context (and an apology)
seancribbs.com
seancribbs.com
Why not? Did the author at least contact 10gen with the test case?
I'm sure that Riak has bugs too - there are very few software products that DON'T (the provably-correct C compiler CompCert[1] being one that shouldn't). But it seemed to me that the post's author was simply trying to avoid getting sidetracked.
A test case that confirms data loss would actually be one of the few things that COULD make this MongoDB discussion constructive.
It's pretty easy to demonstrate data loss with MongoDB if you're doing replication, but it's the "normal" behaviour because MongoDB uses asynchronous replication and W=1 writes by default.
Set up a n=3 cluster and a client that writes data continously, like:
i = 0; while (true) { write(_id:i, data:i); getlastresult(); /* to make sure the client sends it */ printf(i); i++; }
Now kill the master. Another node will become the new master. Issue some more writes to make sure the logs of the new and the old master diverge. Now bring back the old master, which will become a secondary, and it will say something along the lines of "finding common oplog point", and it will discard the writes that it had that were not copied to the other nodes before it was killed.
You can verify all this by looking at the i's that were acknowledged by the old master and printed by the client. The last couple of them will be gone for good.
If this is unacceptable to you, then you can run MongoDB with W=majority mode, but with MongoDB W>=2 modes (so-called consistent replication modes) are very slow.
Serious question: are the consensus modes slower than for any other system (e.g. HBase, Cassandra), or is this just a re-statement of the fact that writing to N > 1 machines is inherently slower than writing to a single machine?
Cassandra does not perform replication/synchronization on a per command basis between the nodes. Roughly: the client writes to multiple nodes, which are mostly independent, so assuming client bandwidth is not the bottleneck, writing to W=2 nodes will not be much slower than W=1. In practice, since Cassandra's disk storage subsystem is also fast for writes, it's overall very fast at raw writes. (As in, fastest in my benchmarks.) The trade-off is that its replication model is eventual consistency, and reads are somewhat slowish. On the other hand, their model works well in a multi-datacenter environment (along with Riak).
MongoDB uses an asynchronous replication model. What seems to happen if you specify W=2 is that the master doesn't ACK the write to the client until one of the slaves has copied it off the master. In my measurements W=2 ran at a fixed ~30 writes/sec on EC2, which means this mode may as well not be there. (This W=2 performance problem was also verified by customers looking at MongoDB.)
If you look at my company's product, ScalienDB, it uses highly optimized synchronous replication model (Paxos) and a storage engine designed for that. It's actually faster running in W=3 mode than certain other NoSQLs in W=1 mode. My bet is that this is what enterprises are going to want if they're going to use a NoSQL as a primary-copy database.
(Test for youself, all products are open-source, it'll cost you less than $20 on AWS.)
2) Cassandra read performance is on par with writes now: http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-...
3) Your explanation doesn't make sense to me. No matter the value of W, MongoDB should make best efforts to get it to all the replicas, no? So lower W should affect availability and perhaps latency but throughput should be unaffected, given a benchmark with sufficient client threads.
Turns out MongoDB doesn't scale well with number of connections due to software issues (eg. one thread per connection instead of async io). At about 500 connections Mongo starts to break on the platform we tested.
The description above left out an important part of the process that occurs in this situation, but it is documented: http://www.mongodb.org/display/DOCS/Replica+Sets+-+Rollbacks . Note that the data that has been rolled back is saved to a file so that it can be applied again if so desired.
As mentioned here, if you don't like that behavior, you can use write concerns, and require W=2 (or more) and wait for the writes to be replicated. Of course, there's a performance cost to doing that, but you can choose.
Truth is ScalienDB runs in W=3 mode at about the speed MongoDB in W=1 mode, so this is not a trade-off that customers have to make.
And yes, these are just different databases. MongoDB is much more RDBMS-like (in terms of consistency, B-Tree indexes and queries), while Riak is what you think of when you think "auto-scalable". So now I love them both :-)
1. evaluate the default settings with which MongoDB ships and
2. evaluate MongoDB performance with more strict,
consistent options and learn that it's quite slow and
3. look at my comment above
Currently 10gen is shooting for light-weight web workloads to spread the product, and they're very successful.
So MongoDB gets it's own niche: use it for "not 100% important data", but instead you get really high availibility of MongoDB cluster, really high speed and document-storage.
What you said about speed on safe commits is really interesting for me. I mean, of course they're slow consequentially, but in parallel they should be just as fast. Or am I wrong here?
You can specify `safe`, which means the data will be written to the journal. You can also specify `fsync`, which means additionally the system will issue an fsync on the journal. Additionally, there's the question of the replication protocol, `W` in MongoDB. What I said is that if you turn on `W=2` it will be slow. You should also know that even with `safe` and `fsync` you may lose data if running in `W=1` mode if the master goes down in your replicata set. See my comment above and the reply from a MongoDB guy.
There appears to be a bug in Riak 1.0, rendering it totally useless, at least with the NodeJS Driver:
https://github.com/frank06/riak-js/issues/97#issuecomment-25...
tldr; Executing a save then remove then a getAll yields a 500 error.
...so we moved our app to MongoDB. Critical mass in the community and the myriad of 3rd party drivers and tools that depend on each other means I'd vouch this kind of issue has better chance of being fixed quickly if it occurred in the NodeJS MongoDB native driver.
In our case, we are yet to have any specific needs of a NoSQL DB, just one that works, and this experience gave me no confidence in the Riak ecosystem. Sorry guys.
- I don't understand this mindset. Forgive me for being naive, but if we're to "toast to the challenges and all become stronger, more proficient, and more successful as a result," then why don't we share these edge cases and fix them?