MongoDB, Data Durability and Improvements coming in 1.8
paperplanes.de
paperplanes.de
Sure, we all know the saying about assumptions. But why are we at a point where it's cool to use a data persistance layer that assumes it doesn't have to live up to any of those assumptions? It's why we have things like ACID, the LSB, etc. Contracts about how these systems should work for the end user.
It's hard to get all that stuff right off the bat, sure. But losing the entire database when a few bytes land in the wrong place? Yikes. Hardly something I'd trust real data to.
This stuff can always get better. Of course the solution as pointed out is tried and true. Glad to see that the developers aren't covering their ears...
Honestly, I would never use a database that doesn't make safeguarding my data as its number one priority. Even if I'm just dumping logs in it, the time I really need to know what's going on is likely the time those safeguards are put to the test.
On-disk, CouchDB never overwrites committed data or associated structures, ensuring the database file is always in a consistent state. This is a “crash-only” design where the CouchDB server does not go through a shut down process, it’s simply terminated.
http://couchdb.apache.org/docs/overview.html
(Please, no mongodb vs. couchdb wars here)
In my case, that meant going back to MySQL once our schema was finalized (sadly). Looks like that won't be necessary in the not-so-near future, which means MongoDB is usable in new projects without worrying about replication before otherwise necessary.
Maybe I was doing it wrong, but it seems like if your app has more than a little data and you plan on changing your mind about how best to present it, you will experience discomfort.
CouchDB supports ad-hoc queries in the form of 'temporary views'. The downside (and I think a difference compared with MongoDB) is that temporary views can't be used in production since they are not indexed and thus are much slower than permanent views.
I'm developing my application in Python, which has great support for CouchDB in the form of CouchDB-Python (http://packages.python.org/CouchDB/) and CouchDBKit (http://couchdbkit.org/). Using CouchDBKit, it was pretty easy to set up CouchDB artifacts (map functions, reduce functions, design documents) into a nice file/folder hierarchy and write the simple Python glue code to deploy updates to CouchDB via a single shell command.
so far we're going with couch and haven't experienced this. the best part of course is that we can add items to documents on the fly whilst keeping the existing test data! this is major for us. we can go back to testing right away. with MySQL the changes were so major that it was better just dumping and recreating from scratch (we are using django) and this wasn't exactly fast. django has a project "south" that would help with this, but we're eventually deploying to oracle and cannot rely on it or have the time to extend it to properly work with oracle.
I have an app I'm working on launching right now, it's in what you might call 'private beta': real people accessing the app is <10. It's been in this state for a little less than 2 months and there's been a replica-set and backups happening since even before that.
What I'm saying is that you don't need to grow before adding a second box. My project currently runs on ec2 micro instances and the second box will probably cost you less than your GitHub code hosting costs! Just seems to me that (in ec2 terms, as an example) going from $9/month burn to $18/month is hardly a show stopper.
Regardless, it's added complexity that you can avoid with MySQL early in your application's lifecycle. Even if it's not much more expensive, you have more hardware to administer. Compare that to a Rails+MySQL stack, which you can run on a single box. (There are added benefits of replicas, sure, but they aren't easier to set up, per se, then just installing your stack on one machine when you're getting going.)
As to the rest, sure, it's sys-admin-wise easier to setup. But then you have to consider stuff like database dumps/backups. Where are you storing them? Have you tested your recovery plan? Then there's hassle when you decide you need a dedicated db box and move it there (friday fat-finger, anyone?). True enough that none of these are deal-breaker level issues (evidenced by the ubiquity of MySQL/etc).
With mongo you spend a tiny bit of extra money and a tiny bit of extra time setting up a replica set which handles your backup for you. And then when you need more db horsepower you just start sharding or just adding more replicas and sending reads there. Need to upgrade the db box? Cool just add a huge instance, add it to the replica set and promote it when it's done syncing.
I certainly agree that it's a trade-off... it's just one that I think is not driven by data durability. Either way I need to handle my disaster recovery with some kind of automated system and I'm just suggesting that whether this is some kind of cron-job, restore recipe, etc backup of a db-dump or a replica on a cheap extra box is a wash.
There are several well-documented, major production deployments of MongoDB already, that seem to contradict your assertion:
* Etsy
* CERN
* BoxedIce (600MM+ documents)
* BuzzFeed (400MM+ datapoints/month)
If your data was worth anything you'd also be using replication in which case the single server durability point becomes moot.
In each of the cases where the customer had a true standby system, implemented via replication, log shipping, or mirroring, they were able to failover with little (log shipping) or no data loss.
In the cases where they had a single, standalone server, the option was to restore the last known good backup, or sent the database files had to Microsoft for analysis and repair.
ANY system (RDBMS, NoSQL, or otherwise), should have a standby replica to prevent data loss. If you data is stored on a single machine, you are doing it wrong.
I don't know about riak, but we are forced to use couchdb at work, for the wrong reasons I think, and I was not impressed. First, its performances are pretty bad: document insertion is slow unless you batch them, but even though it is slower than in mysql, even though the guarantees about durability are certainly not the same (couchdb was not configured to fsync at each write). Building views is excruciatingly slow, and the view engine does not use all the CPU available (even when IO wait is low, so not a simple IO issue). There are bug reports about this issue (cannot find it at the moment). Replication is not very reliable either for large data (where large against means a few GB, so actually not that large).
I am sure we are using couchdb the wrong way at work (out of my hands), and for the wrong application, but its design choices as well as my limited experience does seem to imply couchdb is not adapted for large amount of data, especially one which are written often (there was an interview last summer with D. Katz who said that there were not yet much optimization for large data).
here is the interview of Damian Katz concerning couchdb: http://howsoftwareisbuilt.com/2010/06/18/interview-with-dami..., which mentions that large data is not a focus.
There are some companies which seem to use couchdb for large usage, for example bbc (http://enda.squarespace.com/tech/2010/3/4/couchdb-at-scale-4...). I don't know their infrastructure, but they claim to server 4 billions requests as of 4th march 2010 since summer 2009 on a 32 nodes (16 master, 16 backups). Assuming that summer starts in september to get an upper bound of the traffic, this means 250 rq/sec on average, which is nothing impressive for an infrastructure with 32 machines without more information about what they do. Generally, I would not say much about this kind of usecases, but since it is often advertised by couchdb proponents, the burden of the proof is theirs.
As far as performance, the key with Couch is to keep the view generation speed faster than the insert rate. So if your users are generating less than about 1000 changes a second, you should be fine on a single server.
More than that and you may need to shard / partition, which can be done in a few different way. The leading option is BigCouch: https://github.com/cloudant/bigcouch
I am actually working on simple benchmarks representative of what we do to see if there is something worth submitting as bug issues.
With MongoDB you simply add a cheap second box and use it as at least a replica of your database. When you get tired of "we're updating the site with shiny new code" interrupting your users, you simply make it into a web-node as well.
Here are some reasons:
* excellent documentation
* runs right out of the box
* excellent libraries like Mongoid
* user can easily perform deep queries (e.g person.address.zip = '90901')
* services are available like MongoHQ
If I want a large cluster to handle "big data" Riak or Cassandra seem to fit the bill better.
If I want speed Redis is great.
If I want a schema-less SQL-like (but not SQL) database, MongoDB?
I was able to make a simple test page that told you the IP of the previous visitor who was closest to you. It took me thirty minutes from "hey mongo has some sort of geo support" to that (and I'm not a web dev and was using ATT 3G to read doc). PostGIS took me longer to figure out how to set a lat,long pair in the DB.
I'm not saying raw speed of development is the best measure of a database, but if your #1 priority is the ability to rapidly prototype, I'd imagine mongo is a good fit in your toolbelt.
Sometimes, data doesn't fit a relational model well. In those cases, something like Mongo is a godsend. And yeah, you could de-normalize your data and get halfway to Mongo, but I'd rather use each tool for what it's good at.
I keep hearing this but I've yet to see any good examples.
Your example is not that unusual or even that complicated. What is about it that doesn't work on a RDBMS? You can't just say the solution sucks because it uses more tables that you feel it should.
In this case, I would have less than 200 types, and probably under 1000 Foo records total. Updates wouldn't have been a problem. The pain was much bigger than the upside. Especially designing the forms that would end up creating those rows...
It's not that it doesn't work. It's that it's not a 'this always fits' solution.
There is nothing wrong with denormalized data in your DB and denormalizing doesn't mean you'd have been better off with some random NoSQL solution. Especially considering that the entire rest of your DB is still very much relational data so what happens to all of that when you dump Postgre for Mongo?
That said, I always try to use what's right. I don't care how easy it is to store credit card numbers is... I'll just use BrainTree's Vault product instead. ;)
But still, good to know. I'm bad with views. :[
Personally, Mongo maps so nicely to Ruby that sometimes, I just use the regular driver rather than an ORM. A database that naturally understands hashes and arrays is lots of fun.
Also, it seems like with Mongo, I can often design most of my pages to need just one query, and therefore they're crazy fast. This might be possible with other NoSQL stores too, and I'm trying to get more experience with more of them.
Cassandra is good for 'big data' if you're ok dealing with the repercussions of eventual consistency, and questionable performance characteristics. Not to mention the mediocre community support.
Redis is a fantastic choice, if you're ok with a limited data model and no diskstore option(coming soon though!)
Picking a datastore that fits your data model, access patterns, support needs, etc. can be a daunting task, and simplifying it to this degree just leaves too much out of the equation.
(I'm not excusing trying -9 as a first resort, but if a process is deadlocked (or pegging the CPU in a tight loop) and won't respond to -QUIT, there's not much else you can do.)