Comment by sandreas | Hacker News Reader

Comment by sandreas | Hacker News Reader

Nice writeup... Although I'm missing a few points...

In my opinion a good backup (system) is only good, if it has been tested to be restorable as fast as possible and the procedure is clear (like in documented).

How often have I heard or seen backups that "work great" and "oh, no problem we have them" only to see them fail or take ages to restore, when the disaster has happened (2 days can be an expensive amount of time in a production environment). Quite too often only parts could be restored.

Another missing aspect is within the snapshots section... I like restic, which provides repository based backup with deduplicated snapshots for FILES (not filesystems). It's pretty much what you want if you don't have ZFS (or other reliable snapshot based filesystems) to keep different versions of your files that have been deleted on the filesystem.

The last aspect is partly mentioned, the better PULL than PUSH part. Ransomware is really clever these days and if you PUSH your backups, it can also encrypt or delete all your backups, because it has access to everything... So you could either use readonly media (like Blurays) or PULL is mandatory. It is also helpful to have auto-snapshotting on ZFS via zfs-auto-snapshot, zrepl or sanoid to go back in time to where the ransomware has started its journey.

> Ransomware is really clever these days and if you PUSH your backups, it can also encrypt or delete all your backups, because it has access to everything

That depends on how you have access to your backup servers configured. I'm comfortable with append-only backup enforcement for push backups[0] with Borg and Restic via SSH, although I do use offline backup drive rotation as a last line of defense for my local backup set. YMMV.

0 - https://marcusb.org/posts/2024/07/ransomware-resistant-backu...

Could you elaborate on your strategy to rotate your disks ?

It's pretty simple: the backup host has the backup disk attached via a usb cradle. There's a file in the root directory of the backup disk file system that gets touched when the drive is rotated. A cron jobs emails me if this file is more than 3 months old. When I rotate the disk, I format the new disk and recreate the restic repos for the remote hosts. I then move the old disk into a fireproof safe. I keep four drives in rotation, so at any given point in time I have the online drive plus three with progressively older backup sets in the safe.

And then, after a year what do you do with the oldest hard drive ? Does it enter the cycle again, do you destruct it or do you use it in a failsafe environnement ? The procedure looks OK and I would like to make it more organised myself, just trying to find the right balance.

The drive enters the cycle again. I use the drives until they show signs of failure (SMART monitoring/testing), or until I need to upgrade for capacity reasons.

I'm using "recertified" (really, used) drives that I've written about here: https://marcusb.org/posts/2024/03/used-hard-drives-from-tech.... They are inexpensive and, so far, have been very reliable. (And, yes, I've done restores from the backup sets.)

Thanks for the reference, it makes sense.

Since you mentioned restic, is there something wrong with using restic append-only with occasional on-server pruning instead of pulling? I thought this was the recommended way of avoiding ransomware problems using restic.

There are several methods... there is also restic rest-server (https://github.com/restic/rest-server). I personally use ZFS with pull via ssh...

> tested to be restorable as fast as possible

That depends on your goal, right? If it took me six months to recover my family photo backups, that'd be fine by me.

> So you could either use readonly media (like Blurays) or PULL is mandatory.

Or like someone already commented you can use a server that allows push but doesn't allow to mess with older files. You can for example restrict ssh to only the scp command and the ssh server can moreover offer a chroot'ed environment to which scp shall copy the backups. And the server can for example daily rotate that chroot.

The push can then push one thing: daily backups. It cannot log in. It cannot overwrite older backups.

Short of a serious SSH exploit where the ransomware could both re-configure the server to accept all ssh (and not just scp) and escape the chroot box, the ransomware is simply not destroying data from before the ransomware found its way on the system.

My backup procedure does that for the one backup server that I have on a dedicated server: a chroot'ed ssh server that only accepts scp and nothing else. It's of course just one part of the backup procedure, not the only thing I rely on for backups.

P.S: it's not incompatible with also using read-only media

I don't understand why this is dead..is it wrong advice? Is there some hidden flaw? Is it simply because the content is repeated elsewhere?

On the face of it "append-only access (no changes)" seems sound to me

TacticalCoder's comments appear to be auto-deaded for the last week or so.

I did not see a likely reason in a quick review of their comment history.

You can view a comment directly by following the "... ago" link, and from there you can use the "vouch" link to revive the comment. I vouched for a few of TacticalCoder's recent comments.

Pull-only mode is about reducing that chance of SSH exploits even further.

Reply on news.ycombinator.com