Ma.gnolia.com crashes hard -- no backups?
ma.gnolia.com
ma.gnolia.com
* Replicated Database with one slave in a separate data center just as disaster recovery (not a performance thing with the separate data center). Heartbeat for the slave(s) in the same datacenter so that it stays up.
* Nightly backups offsite. Used to be rsync, but recently thinking of using tarsnap for the ease of it (and S3 puts it in several data centers).
* Files stored either in a MogileFS setup or S3 in multiple data centers.
It doesn't have instant failover to another data center, but the offsite DB slave should mean no data loss beyond a second or two. Ma.gnolia is a decent sized site. Maybe they did have a decent infrastructure and it will only be a little while before they've gotten everything back. Of course, after the fiasco with the blog site that used RAID as their backup system, I've started to think that many people don't take data as seriously as it needs to be taken.
http://www.techcrunch.com/2006/06/29/couchsurfing-deletes-it...
http://www.couchsurfing.com/crash_page.html
Couchsurfing had a backup process in place, but failed due to human error. It's highly likely that magnolia had backups of some sort. Before we all start berating magnolia (and website owners in general) for not backing up their data, we need to know more about the real reason for this failure.
Couchsurfing was able to come back from the dead because of limited competition. In this case, unfortunately, it will be easier for users to look elsewhere.
(Summary: Small hosting company advertising multiple backups; their single server crashed and they had no backups at all. Long story involving terrible customer service ensues).
And human error shouldn't really be possible if your backups are automated and you periodically do test restores.
http://www.rustyrazorblade.com/2008/05/07/mysql-time-delayed...
unborked[.com] is currently available.
A site dedicated to stopping all of the borking going on out there.
===
I think some cheap, simple sql dumps from the db, burned to an external HD and taken home would've been a good first step -- your data would be dated but somehow restorable nonetheless without the need for off-site, replicated, heart-beat monitored databases.
IMHO you really need an expert database admin (of which I am not) to get all of this up and keep all of it going; a caretaker who's only job in life is the maintenance of any and all persisted data.
On another note, backups are really useless unless you have actually tested the restores.
is this some new tech that I'm unfamiliar with ;) ?
Tarsnap lets you say: tarsnap -c -f backup01302009 mysql_dir/
And you can just adjust the date each day. It gives you the luxury of a full dump (anytime you want to restore, just reference backup01302009), but it only actually stores the deltas (making sure not to duplicate data that might be in backup01292009 or backup01282009 and so on). Tarsnap stores the data to S3 so that it's replicated in multiple data centers.
It costs a little more than S3 at 30 cents per GB, but it's metered out so that if you only use 1MB of storage, you'll only be charged 0.03 cents for that storage. You could try creating your own way of doing incrementals, but I doubt you'd get it as efficient as Colin (the math genius behind Tarsnap) and so I doubt you'd get it cheaper. Plus, this way you don't have to deal with it.
And remember, it's hard to fill up a database.* As the Django Book notes: "LJWorld.com's database - including over half a million newspaper articles dating back to 1989 - is under 2GB." So, if they were using Tarsnap, they might be storing 5 or 10GB tops at a whopping $1.50-$3 per month plus whatever the transfer of their deltas was for the month. Oh, and tarsnap compresses the data too. So, maybe they'd be paying $1 or something lower.
* Clearly, if you hit it big time, you might not want to continue paying for tarsnap. However, if you become the next big thing, you can hire someone to deal with it for you.
You either need to a) use InnoDB hotbackup or b) use a slave, stop the slave, run the backup, and restart the slave to catch up.
At delicious we used B, plus a hot spare master, plus many slaves.
Additionally, every time a user modified his account, it would go on the queue for individual backup; the account itself (and alone) would be snapped to a file (perl Storable, iirc.) Which only got generated when the account changed, so we weren't re-dumping users that were inactive. A little bit of history allowed us to respond to things like "oh my god all my bookmarks are gone" and various other issues (which were usually due to API-based idiocy of some sort or another.)
You use the slave so you can shutdown the database and get a consistent file snapshot. Then you do offline backup.
I use this for my personal backups, as well as backups of our work svn (fsfs) and git repositories. I use it against S3, and have found it incredibly reliable.
As a bonus, it encrypts everything but still does incremental backups. It's a really nice piece of software, and you don't have to pay anyone to use it.
Duplicity is not the same thing as tarsnap. Duplicity uses a full plus incrementals model compared to tarsnap's snapshot model, so with duplicity you're either going to be stuck paying to store extra versions you don't want or be stuck paying for multiple full backups. Moreover, tarsnap is considerably more secure than duplicity.
Before I started working on tarsnap, I considered using duplicity; but it simply didn't measure up.
1. Duplicity uses GnuPG. GnuPG has a long history of security flaws, up to and including arbitrary code execution. Yes, these specific bugs have been fixed; but the poor history doesn't inspire much confidence.
2. Duplicity uses librsync, which follows rsync's lead by making rather dubious use of hashes. In his thesis, Tridge touts the fact that 'a failed transfer is equivalent to "cracking" MD4' as a reason to trust rsync; but now that we know how weak MD4 is, it's possible to create files which rsync -- and thus Duplicity -- will never manage to back up properly.
3. When you try to restore a backup, the storage system you're using can give you your most recent backup... or it can decide to give you any previous backup you stored. Duplicity won't notice.
4. If you try to use the --sign-key option without also using the --encrypt-key option, duplicity will silently ignore --sign-key, leaving your archives unsigned. Based on comments in the duplicity source code, this seems to be intentional... but this doesn't seem to be documented anywhere, and it seems to me that this is an incredibly dumb thing to do.
Beyond that, drive snapshots aren't the easiest things to do. I know that Right Scale tells their customers to freeze the drive so that no changes can occur until the backup is complete. With S3 performance around 20MBytes/sec, to backup 1GB would take around a minute. That's not bad and only doing deltas it's unlikely you're going to have a huge amount to backup at any given time, but it isn't exactly good either. With file-level backup, you can do a mysqldump and then just back up that file. Eh, maybe I'm just preferring the devil I know in this situation.
It's a little more complex to set up (doing file-level backups), but if you're going the volume route, you need to make sure you don't leave the drive in an inconsistent state.
All that said, EBS is awesome. If it fits what you're looking for, then go for it!
We snapshot a slave every 10 minutes and the master once a night (just in case something totally weird happens to the slave and the sync isn't right). This is a multi-gig DB and we've had no problems.
Here is a link to a full tutorial about running MySQL on EC2 with EBS: http://developer.amazonwebservices.com/connect/entry.jspa?ex...
I wanted to also point out that a live slave is NOT a backup scheme. If someone hacks your database and runs DROP ALL FROM PRODUCTION_DATABASE you've now got a perfect copy of nothing.
Disk space is cheap -- I do a full dump of the database nightly (and a separate system dumps a few key tables every 15 minutes)
If you don't have a backup plan -- close your browser, stop surfing Y!Hacker, and go write a shell script to dump your database and rsync to anywhere else on earth.
We talk to a lot of customers about their backups every day. I'd say more than half of the backup failure stories we hear come from failing to adequately test that data can actually be restored.
Some have been comically tragic. Like creating an offsite backup using encryption keys that are only archived as part of said backup... :(
Ugh, that is awful.
then once they get the point they want to leave the room asap...
Run the dump file every N units of time.
Compress the dump file using gzip with the --rsyncable option. This increases size by 1% but makes it efficiently diff/patchable.
Use diff to make a patch going back to the previous compressed dump of the file. Keep the patch, discard the previous dump if you like. You now can apply the patches in succession to go back however far you like.
Finally, and most importantly, use par2 to store parity along with your backup files to protect against silent bitrot.
Note: --rsyncable is in newer gzip versions. It's too new to be included in OS X Leopard.
* Backup the Binlog of the DB (At least its called Binlog in MySql)
or
* Make a diff of the uncompressed dump and zip that. This diff would even be human readable.
Unfortunately, it's not enough just to have backups. You have to actually verify that they're correct and up-to-date. Verification is easy when your database is small (for example). You can just load it in your development environment occasionally. But, how do you verify your backups if you have hundreds of gigs or even terabytes of data?
As an example, I've seen cases where backups were successful every night... but, they were being run against a slave db and replication had failed. The result: Excellent backups of weeks old data.
A disk raid is only a raid when you have tried pulling a disk and having the raid successfully rebuild (while running in production!).
I once had a system administrator who didn't dare pull a raid disk while in production. After that, has wasn't a sysadm in my eyes, he was a lottery gambler.
Being paranoid doesn't mean nobody is following you.
That's the thing with backups, they can fail or become corrupt in many ways. If you don't use them for something on a regular basis or have some regular verification process you may not know what you have until it's too late.
And, of course, I've also seen situations where subtle data corruption in the master database leads to weeks of subtly corrupt backups. By the time the corruption was discovered we faced the choice of rolling back several weeks to the last good backup or fixing the corruption. In the end we had to do a kind of merge -- it was a real pain.
As a baseline, you should at least restore from your backups occasionally.
It helps that I'm familiar with the data -- so after restoring the backup I expect to see games, messages, forum posts, etc that I've just seen in production.
I do some more thorough automated tests on the backups less frequently. App specific things, like replay games and verify results, etc. This process is more about assuring backward compatibility with the code though.
I think verification should ultimately be somewhat app specific. That said, I'm sure you can find tools to help with verification.
Our motto is "If it's not backed up, it's down." (Applies doubly so for us as a backup company.) For sufficiently seasoned administrators, non-redundant storage should cause sleeping difficulties.
The first thing I ask when my boss tells me to set up something new: how are we going to back that up?
This runs counter to the more cavalier "release early, polish later" advice I keep seeing. Maybe I am doubly freaked out because the things I'm storing are not easily recovered or re-imported by the users themselves or any kind of algorithm/redux.
Doesn't mean it's a bad idea though. But if it's that good, make sure you announce it and market it as a significant feature...
I want to not only have a backup scheme but also make sure it's restore-tested. Maybe I wasn't totally clear, not planning on a beautiful failover in each place in the beginning (planning failover for the DB at least). Just a tested (even if manual) restore procedure in each situation.
I suspect their file storage system is simply good enough to replicate data intelligently across many machines/sites like a giant RAID array.
I also think that it's a truly massive benefit they have over other companies, particularly small companies, and that it's rarely discussed as such.
I agree with you that that is one of Google's key strengths, though. PageRank is neat, but the real edge is that the site is fast and the infrastructure is based on commodity hardware, so it's cheap.
It wouldn't matter if they were blades or Sun UltraWhatevers, the real benefit is that if one dies, nobody need panic - plug another in and "the system" will rebuild it. If a rack dies, plug another in and the system will rebuild it. If a datacenter dies, the others will cover for it.
No backup tapes, no manual tweaking each new build.
Adding a few more servers is much cheaper than downtime.
Just had 3 days in a row over 43 (109) and todays pretty warm as well. (my house still hasn't cooled down for sure)
Maybe I'm just lucky enough to work with relatively small datasets, but our QA server contains a complete copy of the production database and is at most a week old. So even if we didn't run any other backups (which we do), we could restore from that server and be back up and running within hours.
It seems like a convenient re-assurance for users to get DVD backups mailed to them every so often -- but it would be a total pain to support this as a small business...
At about, say, 2K per bookmark, that's a couple million bookmarks. If one bookmarks a site every 60 seconds, 24x7 that's about 4 years of work. A slightly more reasonable person, bookmarking every 60 seconds for only 8 hours a day, would have to be bookmarking websites continuously since about 1996 to fill up a DVD.
But how do I offer this when we're in colocation or slicehost etc.? Can't be driving to the colocation center every time someone wants a DVD...
Of course photos and whatnot are much harder.
I broke this out... http://news.ycombinator.com/item?id=458921
The way I see it, there are too many social bookmarking sites out there. If there's now one less, I wouldn't mind.
This does not require a lot of sophistication. And yes, we might have some downtime while we restore in the event of a catastrophe, but we do have the data.
I can't even imagine the nerve of shipping a product to the public that doesn't have even something this rudimentary in place.