Hacker News Reader

Comment by mike_d | Hacker News Reader

People often forget the 3-2-1 rule should also apply to the secrets you use to encrypt/store your backups. If you can't decrypt it, it isn't a backup.

I use a crazy long passphrase to encrypt my backups, but should I forget - it is also printed on archival paper inside a sealed envelope in a friends safe deposit box (I also have a copy of his backup passphrase for mutually assured destruction :)).

Also, every once in a while run a fire drill and actually restore something from each of your backups. This is when you find out the rsync job has been stuck for the last 80 days.

View on HN

Yeah the fire drill to make sure everything works is crazy important.

I just wish that storing things like keys on paper was easier.

At CoreOS we put some keys on printed QR codes and scanned them with an airgapped laptop every 90 days to confirm the keys were safe.

> Yeah the fire drill

I agree. But I've never been allowed to run a fire drill. Rebuilding a network from bare tin is obviously expensive, but not as expensive as losing the business.

And then there's the sheer stress of being responsible for the backups, but not being able to test bare-metal recovery.

I'm interested in backup ("what kind of weirdo is this!"), but that wasn't a fun responsibility.

> But I've never been allowed to run a fire drill. Rebuilding a network from bare tin is obviously expensive, but not as expensive as losing the business.

Sometime your job is not to be able to take the backups, but to take the blame when a restore can't be done. If you think that might be you, time to polish up that resume...

Especially for cloud-based businesses, refusing to let you spend ~8 hours of production platform costs to run a full platform rebuild fire drill is insane, that's about 0.1% of your annual prod AWS budget, and 0.5% of you (or your team's) annual time.

We weren't cloud-based; "bare-metal restore" would have meant acquiring and housing new hardware. And we'd have needed test-scripts and so on - more than 8 hours, more like a few days to do it repeatably (but once that was done, it could have been automated).

The boss owned the company; I think it was shortsighted of him. Being able to blame me wouldn't have done him much good, once his firm had gone down the tubes, and I'd moved on. But it was his business to lose, and he knew what his margins were - I didn't.

He didn't like me much, mainly because he was a control freak, and I knew his systems much better than he did - I had the control that he craved. Control freaks shouldn't hire people that know more than they do.

You could use Samir Secret Sharing to split the secrets in M unrelated strings and then "recover" it with M-k strings. (E.g. Generate 5 keys and require 2 to recover). That way you can share some of those keys but nobody will have the target password.

If I am in a situation where my brain doesn't work well enough to remember a backup passphrase, it isn't going to work well enough to do silly hacker shit.

There are very clear directions to my friend for when it is acceptable to access my stuff, and very clear consequences for malice.

You can also write very clear instructions on the recovery messages. I plan on doing this very thing for my disk decryption passphrase and password database. It's a standard package available in the base repos, and I'll have step by step instructions on how to recover it. If they can't follow the instructions, then they probably shouldn't be trying to handle my affairs in the event of an emergency, either.

> I also have a copy of his backup passphrase for mutually assured destruction

I have two friends who've given me private key fragments (from a Shamirs Secret Sharing setup) with the explanation "If anything happens to me, you'll work out who you need to talk to and what you'll need to do."

I haven't done that myself, because I don't have any need/desire for anybody to decrypt my backups once I'm not around. That might change if I end up with dependants one day.

For your last point: That is the reason I implemented https://healthchecks.io/ into my jobs.

Fire drill would still be a good practice, though: during the fire drill you can find out your backup is corrupted, or is incomplete (you are not backing up everything you thought you were), or inaccessible (you encrypted the backup, but where are the keys...).

He never said fire drill was not important?

> This is when you find out the rsync job has been stuck for the last 80 days.

Fire drills are important, but I get a Telegram notification every day when my rsync jobs complete. The notification is muted, but I see it in the list every morning when I use Telegram to text my girlfriend. If it wasn't there, I'd immediately know something wasn't working.

A full test is absolutely needed - including any system or DB restores and the system needs to be fully functionaly tested

A quick backup verification is not enough - I learnt this the hard way and lost 9 months of data only after I did full annual DR test. The backups were setup and configured by the (very well known) manufacturer of the backup software. They screwed up on the DNS name

> I use a crazy long passphrase to encrypt my backups, but should I forget - it is also printed on archival paper inside a sealed envelope in a friends safe deposit box (I also have a copy of his backup passphrase for mutually assured destruction :)).

Why not just remember the password and perform a regular fire drill decryption to ensure you won't ever forget it?

For backups that I can safeguard physically (as opposed to cloud), I just copy files uncompressed and unencrypted using FreeFileSync and Syncback. Restoring now is as easy as media swap.

I prefer the "1 rule" for some data... the password is only on my mind, so when I die, it's hopefully gone forever :)

While I have some passwords that I don’t think I’d ever forget not sure I’d want to bet on it given that I’m remembering more than one password.

I have data like that... however there is some data that needs to survive me for my family / estate. Proper planning for that is something everyone should do.

I have been involved or aware of a few nightmares where an expected loss resulted in all kinds of pain and problems for the family has they lost access to critical accounts, financial data, and other things that required lengthy interactions with government agencies, banks, etc that would not have been needed if the data would have been able to be recovered from the systems

You think you have his password...

To be honest I wouldn't have mentioned that on a public forum, but it's kind of you to have done so.

Reply on news.ycombinator.com