Ask HN: How do you backup your servers? | Hacker News Reader

Ask HN: How do you backup your servers?

8 pointsbmaeser6 comments

I do a lot of devops-work for different companies and "agencies", and i have seen a wide range of solutions so far. Unfortunately most of the existing backup solutions were really bad an i realized how less they even cared about it.

So HN: how, what, how often, where and when do YOU backup your servers?

examples:

configfiles, everything at /etc/ is under versioncontrol with git, everytime i make a change i commit and push it to a private server (offside) where every server has its own repo.

database, postgresql, every night at 02:00 a full backup with pgdump, triggered by cron, homebrew backupscript in bash, dump gets scp'ed to another host in a different datacenter, old backups are deleted manually when the disc gets full soon.

View on HN

I used to use a homebrew script that implemented rsync/snapshots (that is, rsync data to a directory on another server, then "cp -al" all files to create hard links into a directory called "server-datestamp-weekly" (or daily, monthly, etc).

After running into scaling issues (esp. when needing to find a specific file that I didn't have a date or server name), I wrote (and open-sourced) Snebu. This works effectively like rsync/snapshots, but it stores the file metadata in an sqlite database (so there is very little setup complexity / maintanence), and individual files are stored compressed in a vault directory, named by the SHA1 checksum of the file. This gets cross server file-level deduplication for free. This is somewhat similar to how git stores files.

Just recently posted a 1.0 release, although I need to improve the documentation a bit and post a few more front-end scripts (and finish the in-app help pages). But would love to get some feedback on it.

The site is www.snebu.com if you are interested (this points to the Github-hosted page, the project page is github.com/derekp7/snebu).

For data retention, I keep daily backups for a couple weeks, weekly backups for 6 weeks, monthlys for a year, and yearlys for as long as I have space.

great project! so you call snafu still from custom sh-scripts and trigger them by cron?

how do you backup your databases?

Actually, it is "snebu", "simple network backup utility", but it does come in handy in case of a snafu...

For database backups, I'm only doing Oracle DBs right now. The procedure there is: 1) Back up all non-database files 2) Put the database in backup mode 3) Backup the .dbf files 4) Take database out of backup mode 5) Force a logfile switch 6) Backup the archive logs (archived redo logs) I'll have to make sure I include a sample Oracle backup script (along with other DBs if I get a chance to) either in the documentation, or in a samples directory.

So one of the features of Snebu is that if you re-use a backup name / serial number (datestamp), it will append those files to the existing backup. As for scheduling the backups, on my backup server I have a cron job that cycles through all the hosts to backup up, and kicks off the client-side backup script via ssh. So: backup-server -> runs master backup script, ssh's to target host target-host -> runs backup.sh, which calls snebu via an ssh command to the backup server

By having the master cron script run from the backup server (or actually any centralized host), you can keep the backups from overlapping each other. Currently, due to the way sqlite handles locking, running backups in parallel can be a bit problematic (although I think I've got most of the concurrency issues resolved, still have to QA my code changes though).

oops! VERY unfortunate typo. sorry for that. i think sigmund freud would have its own thoughts on it :-)

i really like your approach, i have very similar setups (backup-master && remote execution via ssh) and put a lot of thought into this topic the last couple of weeks. i think there is way to less love and tools who properly handle backups. everyone runs his own scripts and tools, or buys some extremly expensive and proprietary "stuff".

the whole point behind this thread was to get insight on other peoples system, look at the pros and cons and write a library that fits 90%+ of the use cases.

as you can see by the amount of people, participating in this threat (by now), no one gives a shit. which brings us back to snafu.

thanks for the insights and "snebu" thou!

That's ok, you've given me a good tagline -- "Snebu comes in handy when you have a Snafu". As for comment participation, unfortunately unless you luck out and get a few upvotes immediately when you post here, you won't make the first couple pages and no one will see your submission. I had the same issue when I posted a "Show HN" about Snebu, no one saw it. I think the only way to get a submission to be visible is to have a few friends up-vote you right away (about 3 - 5 is all it takes to briefly make it to the front page), then if enough other people like the content it will stick.

Chef

Reply on news.ycombinator.com