Bare metal.
Frequently changing data.
Occasionally changing data.
So I have a bare metal backup of a compressed image of my system drive (made with dd using a live-CD) which is updated every time I mess with LVM and nightly snapshots of my root partition (LVM snapshop of partition with dd).
From that toe-hold I can rebuild my system from any worse case short of hardware destruction, I use nightly backups with BackupPC for normal system recovery, augmented with what's below for frequently changed data.
Frequently changing data, like email and code I'm working on, is rdiff-backup snapshotted every hour. If I was maintaining customer data I'd do a continuous process.
The same data is rsyncsrypto-ed and most of that is rynced to rsync.net.
Occasionally changing data is rsynced every night.
I try to avoid common mode failures, e.g. no one piece of software, one system or one disk will cause total data loss.
Ask if you want particular details; for a long treatment of this, read Backup & Recovery, "Inexpensive Backup Solutions for Open Systems" by W. Curtis Preston (http://www.amazon.com/Backup-Recovery-Inexpensive-Solutions-...). It's getting to be a bit old, but a lot of it still holds and the principles will be true for a long time.
(Note, while the above particularly paranoid approach is for my personal systems, it's based on harsh experience going back to 1977 (sic), including plenty of small scale "enterprise" systems where total data loss was unacceptable and a couple of 24x7 Internet startups.)
Cron + snapshots + small custom script for EBS on EC2.
Basically, I store all non code files on EBS and redundantly backup to S3, which amounts to living or dying by Amazon's hand.
Recovery consists of firing up a new EC2 instance, attaching a snaphot and running a deploy script, which is handled automatically by a different EC2 instance, assuming I haven't lost all instances. Manual recovery in that case would be time consuming, but doable.
Snapshots are nearly instantaneous and uploads to S3 from an EC2 instance are about as fast as you can get.
Most annoying thing? Testing recovery.
Indeed ... but at least you do that!
One of the great things about clouds like AWS is that you can now do an almost complete bare metal restore test (modulo stuff like your official DNS vs. test) for a small amount of money instead of trying to justify buying a separate system for that sort of thing (which in the real world I was never able to do, although I fortunately never suffered a total data loss despite a few close calls).
On my other servers I use rsync just to keep a current mirror around. I always ask myself, "How hard would this be to recover if I lost it?" and "Will I need to go back in time for some things?" Depending on the answer to those questions I make my backup strategy for the server in question.
Yesterday's Daring Fireball post on the same topic said it best: "Every hard drive in the world will eventually fail. Assume that yours are all on the cusp of failure at all times. It’s good to be spooked about how long your hard drives will last."
Of course it only provides for the raw data-transfer task so one has to think about the scheduling and planning yourself.
An acquaintance with decades of experience in the hard disk industry including a fist full of patents including 2-3 in every disk drive you own says:
"One should start planning the death of the drive, and all one's data, beginning the day one buys the drive."
Critical user data, databases, etc. are dump'd/tar'd/gzip'd to a remote location once a day, generally held as long as space permits.
Gold Images: of the various OS configurations and requisite software so you can redeploy a server fairly quickly
Diff Backups taken nightly of our various data stores / databases
Weekly backups of main server data (MySQL, MSSQL, Hadoop clusters etc)
In terms of software we use a couple of commercial solutions plus Bacula (which isn't really that great unless your using tapes...).
Inspired by Tarsnap (the encryption and so forth) Im in the process of speccing up an idea to write out own in house version (I'd love to use Tarsnap directly but not a chance unfortunately) to replace our mish mash of software.
Tar-like interface, data deduplication, awesome support, fair pricing. What more could you ask for?
You forgot security. :-)
Any code / changes to Debian's default /etc setup go into Git and are at least on both the relevant machine (laptop or server) and GitHub. See http://github.com/avar and http://github.com/avar/linode-etc for the /etc on the server. I keep some personal data (like finance notes) in GPG encrypted files in private Git repositories.
E-Mail is backed up my Google's GMail (presumably) and data like my small collection of music is something I can always get back from the Internet.
If the filesystem on my laptop, server or GitHub is destroyed right now I can get everything working again relatively quickly just by installing Ubuntu or Debian from scratch and copying my home directory, /etc and code from Git on one of the remaining machines.
I've made a script that automatically backs up my Git repositories daily (http://github.com/avar/github-backup), the resulting backup is around 180MB. Losing some of my other data would be an annoyance but this is all I /really/ care about.
Follow somebody else's advice if you need "real" backups, but for my simple needs this suffices.
Right now its only done weekly, as we're not experiencing high growth. We also store downloads, so the tars can get very big, but it works well.
I never upload code to the internet, but I have copies of the repository in multiple locations separated by at least 3000 miles.
Nightly I backup internet databases to a cloud storage provider. Nondatabase files are stored simultaneously on local storage and cloud storage. The local storage is mirrored to local snapshots hourly and nightly in case a roll back is needed.
I like having a complete copy of all user-data on my home machine for development purposes as well as backup. So, I rsync all user generated content (they are 99% flatfiles so this is easy), as well as a daily mysql dump. My other founder and I both perform this daily, so at any point we are only exposed with a 1 day risk (seems acceptable at this point). This also gives us two separate off-site backups, which is nice.
when the server boots up, i have to enter in the softraid passphrase, it mounts the drive, then i launch screen and start up the script that spawns the rsnapshot processes at the appropriate times. when starting it, i enter in the common ssh key passphrase (which is restricted on all the servers to only allow logging in from that network) and common mysql backup user password. these are stored in memory and then the script runs rsnapshot every few hours.
once a day, the script does a full mysql dump of all databases on all servers using a ruby script with net::ssh to tunnel mysql access over ssh. it does this to keep "proper" mysql backups in case the rsync'd binary mysql files are transferred in an odd state, and keep daily backups of everything. it also lets me quickly get at data with standard text tools instead of having to restore the raw mysql files somewhere temporarily, start up mysqld, and extract it.
so once the server is booted, it's completely automated. i've been doing backups this way for a couple years and have recovered files and databases from them a number of times. no complaints.
I have a cron job that dumps the databases into a directory with a time-stamped name. The last few regular nightly back-ups are kept, then weekly for a couple of weeks, then monthly for about a year. The parent directory is then backed up to S3 using duplicity. I've done other, cleverer things in the past with rotating through buckets based on the month, keeping every backup, doing them twice a day, that sort of thing.
I test recovery by spinning up another server, running the scripts to bring it up to production status, and running the script that uses duplicity to pull the data dump directory back out of S3, and then restores the database from the dump.
At $0.15 a gig to store data, and about $0.10 a gig to transfer it in, the cost of S3 is pretty much just noise. I've got on-disk dumps of my database if the database is corrupted on my machine, and I've got back-ups of those dumps if I need them.
Duplicity is pretty fantastic -- definitely check it out.
Then we use cron and s3sync to mirror the relevant portions of the filesystem to S3.
For those interested, the process is documented here: