Hacker News Reader

Comment by CGamesPlay | Hacker News Reader

I never really liked the 3-2-1 rule because it feels too specific: while it works, simpler solutions also provide the same level of reliability.

I think about backups in terms of blast radius. 1) The local machine has the working copy of data and a local backup as permitted by free space. The smallest blast radius where I lose data is "my laptop hard drive fails". 2) My external drive has another backup. The new blast radius is "my house burns down". 3) I maintain a cloud backup. The new blast radius is "a catastrophe on a global scale".

Any two of these backups can fail and the data is still salvageable.

View on HN

> while it works, simpler solutions also provide the same level of reliability.

It exists and is important because many backup strategies are broken and people don't realize it.

For example your own strategy treats a PC's local storage and an external drive as distinct backups when in reality you've only evaluated hardware failures when formulating it and not malicious actors. In 3-2-1, in particular the media type + off-site thing, tries to "trick" you into having a backup which isn't accessible from the same system that it is backing up (i.e. offline backups).

Your backup strategy has been used almost verbatim by multiple institutions who got cryto-locked. The external drive was hit and then the cloud backup service happily synced the now encrypted files. They went from "backed up in three places" to backed up in zero places, and are now calling the cloud provider hoping that their backups had unencrypted copies in them.

3-2-1 isn't simple, but it is good, and that's what it tries to be.

In my DIY backup setups, I'm relying on rsync.net's snapshots [0] and also the fact that the AWS S3 bucket I'm automatically copying stuff to has 'versioning' enabled. [1]

Is that good enough in your point of view?

Thanks

[0] https://www.rsync.net/resources/howto/snapshots.html (not affiliated, just a happy customer)

[1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versio... (ditto)

For automated ransomewhere this works.

For targeted - it often doesn't because often your AWS keys are on the system doing the back up and have permissions to delete items etc.

Of course, this is why S3 allows you to set and object lock rule (ie, 30 days is plenty) that means even you (or your computer) can't go and delete those online backups.

"For targeted - it often doesn't because often your AWS keys are on the system doing the back up and have permissions to delete items etc."

Just a reminder - snapshots at rsync.net are immutable.[1]

Even if Mallory gains access to all of your access credentials, she cannot delete/change the snapshots in your rsync.net account.

[1] https://twitter.com/rsyncnet/status/1453044746213990405

I have send internal drive that gets an rsync of my first drive. First drive is xfs currently, second drive is btrfs. After the rsync, the drive gets a snapshot, then it's unmounted.

For cloud, I use a time4vps storage server. I rsync some data and use borg for more sensitive stuff.

That's great to hear about the object lock rule, thanks a lot. I'll add that to my S3 setups.

Also, if this were for a company I'd suggest to test the backup at least once during the object lock duration: e.g. if the object lock is set to 30 days, you have to test them at least once (e.g.: day 15).

That's because a targeted attack with a ransomware could gain access to your servers and wait 30 days while silently encrypting your backups until the 30th day, when the attackers could just complete the attack encrypting the rest of the files and showing the message.

So my recommendation for extremely critical data would be: 1) Test whole data thoroughly at least once during the object lock period. 2) Setup an automatic task that retrieves X random data every day (or the longest period of time you can afford to lose it) and perform checks with checksums or other methods. If something is corrupt and/or encrypted you will realize before it is too late.

Thanks a lot! I appreciate the insight

Backup versioning is the important distinction here. A backup on its own is only good against loss, but a backup with multiple snapshots in varying states in place protects against the threat of crypto-locked cloud synchronization.

Of course if the provider is locked then it's a moot point.

Some backup versioning schemes are disabled on purpose by some cryto-malware.

What you're saying is that a critical piece of the 3-2-1 rule is a piece that isn't actually prescribed by the 3-2-1 rule: that you have "a backup which isn't accessible from the same system that it is backing up (i.e. offline backups)." Another reason that I don't think this "rule of thumb" as useful as it purports to be.

Presumably the rule was invented before ransomware was a thing, so perhaps it gets a pass for not anticipating versions, but yeah: the rule of thumb for the modern world probably includes something about backup versions.

It should be known as the "3-2-1 rule with knobs on" for that precise reason, not to mention several others. However, you need to start somewhere with a pithy pitch.

Veeam for example (other backup systems are available) have several "immutable offerings" whereby you get backups that can't be deleted for a while. These are not appliances but configuration recipes. One of them is a generic Linux box with XFS with reflinks, a particular set of file perms and what they call a one off credential (a sort of app password I think), forward incrementals and a few other things. Get it all in line then even if a baddy gets in they can't delete your backups for a while. This does require detection inside say a week. The reflinks thing is really useful for storage space and things like "synthetic fulls" where you roll up incrementals into the last full backup.

You can also use AWS etc object storage for immutable. In all of these things you can lengthen the timescales available by applying more cash.

For me: tape in a safe is the only decent way to protect against ransomware. Even then some tape hygiene is needed!

IMO it's a stupid rule, but it's not for us, it's for people who don't even have this.

I think there's an element that is as important as keeping a backup safe against a state-wide wildfire and that's AUTOMATION. If it isn't automated then chances are that your backup is very old once you finally need it.

Also need monitoring then!

After the data center fire in France last year, where a lot of people lost their cloud data, including a friend of mine, I also make sure that I've got multiple cloud backups in distributed geographical locations. I recommend looking into Storj DCS.

> simpler solutions also provide the same level of reliability.

far more simpler solutions exist that don't provide anything like reliability.

Things area more nuanced these days than when this "rule" was first formulated, but I suspect it's still true that the vast majority of people would be far better off with following it than whatever they are doing now. Doubly true of personal use.

Yes, this is exactly my point. I think we need a "2.0" version of this rule that better encapsulates what a good backup strategy is.

You missed a radius there, with neighborhood or town/city level disasters.

I only use cloud backups (Glacier and Google Workspace), I gave up on off-site drives as they end up too far away to be easily/consistently updated, or close enough they are in the same disaster area I am in (Earthquake zone).

Houston flooding is another good example.

Am I missing something or is your system actually 3-3-1? I.e. more comprehensive than the 3-2-1.

Do you count "laptop hard drive", "external hard drive", and "cloud hard drive" as different media types? What if the laptop happens to be SSD but the external drive is magnetic? What if the cloud drive is SSD or magnetic? That "2" in 3-2-1 just doesn't make sense in today's world.

Reply on news.ycombinator.com