Hacker News Reader

Comment by ahazred8ta | Hacker News Reader

A gentle reminder to actually TEST your restore workflow.

WBD: since 2011! https://hn.algolia.com/?q=worldbackupday

People always say that but what do they actually mean? Restore one file, check if the backup is bootable in case it should?

I'm sure there will be plenty of advice for testing the backups themselves, but for online backups (either to server or to a service), one thing which is easy to forget to do is to check you can still get access from "scratch" - ie if you have lost all local data. Make sure you don't need eg passwords or keys for the online backup service which you have no other copy of (doesn't help if they are in the backup itself!) or indeed for email/cloud accounts which you might need to regain access to to get to your backups.

For static files like photos I hash check drives against each other to check for bit rot. But yeah, for tape based cloud storage I can't think what else to do except restore one file to check I still have access. It's too expensive to restore the whole archive.

Do you have any automation or scripts for that, or is it usually adhoc?

Yeah, I use rsync with `-cavin`, if the output includes `>` or `<` then there are differences in checksums.

  #!/bin/bash
  hostname=$(hostname)
  checksumFile=/tmp/checksum-file.txt
  
  rsync -cavin --info=name2 --no-perms --no-owner --no-group /srv/data/photos/ user@remote:/media/data/rsnapshot/daily.0/srv/data/photos/ > $checksumFile
  
  minimumSize=5000
  actualSize=$(wc -l <"$checksumFile")
  
  if [ $actualSize -ge $minimumSize ]; then
      fileDiffs=$(cat $checksumFile | grep -e '<' -e '>' || echo 'all checksums match')
  else
      fileDiffs=$(echo $checksumFile is too small)
  fi
  
  echo $fileDiffs

You absolutely should NOT automate this. If you do, you then need to manually check that automation with the same frequency- so you haven’t gained anything!

I do have this automated, but the checksum output is posted to a kind of watchdog service. Every day I get an email that says everything is as expected, or not.

I need to google every time to find a good process for hashing files and comparing across disks? Not only that, but remember to do it frequently enough?

You can also restore the entire backup to /dev/null (this device has lots of free space) or pipe the output into a checksum generator. Borg also has a consistency checking command.

You’d actually want to think a bit more. Where is your paper decryption key? Is it off premises? Test a, some or all of the files in the archive. Boot into an ISO. Try it again. And that for all of your backups. It’s not a backup until it is tested.

Reply on news.ycombinator.com