Money quote based on trying to restore their MongoDB during the EBS downtime:
"Waiting for indexes to build takes approximately as long as it takes AWS engineers to fix EBS issues."
"Waiting for indexes to build takes approximately as long as it takes AWS engineers to fix EBS issues."
I just did a test dump of a 1.7GB MongoDB database:
real 0m5.781s
user 0m0.570s
sys 0m4.817s
A mysqldump / restore can be dependent on many variables when discussing the rate of import. Grouped indexes, delayed FK constraints, buffer sizes, and hardware specs to name a few. My point was more about how one is going to suffer this reindexing problem regardless of data storage facility :)
[1]http://www.percona.com/docs/wiki/percona-xtrabackup:start
In any case, mysqldump is rarely the bottleneck - restoring is. For the latter, you really want to use mysql from percona, because their version of mysqldump has an option to void the data without index first, and create the index later (--innodb-optimize-keys=True). The effect will depend on the database schema of course, but I often see one order of magnitude difference for simple, large tables with a few simple indexes.
You can still write to your database, yes. There's a flag, --oplog, which when passed to mongodump, will record the oplog position at the time the backup starts, then dump any ops since the backup started as a part of the backup, so they get replayed on restore. This leaves your data consistent with the database state at the end of the backup without the need to issue a write lock.
That may be true. Snapshotting is definitely superior for disaster recovery regardless of your database. I hope I stressed that enough in my talk as well as in my slides.