Hacker News Reader

Comment by likecarter | Hacker News Reader

A crucial point you missed:

Ransomware gangs often destroy your backup infrastructure. So it's important to create pull-only backups or backups that cannot be deleted / overwritten.

View on HN

I thought this was sufficiently implied with "keep them segregated from the rest of your infrastructure," but yes, you are correct. It is important to have a set of backups that can't be destroyed by the attackers. You might get lucky that your hand-rolled solution is so hacky that a ransomware gang overlooks it. But better to not rely on that luck.

In the past, this was achieved by having a set of tapes offsite. Today, one might configure Veeam to lie when issued a delete command, and instead send the data off to an Amazon glacier instance that requires different credentials to read, write, and delete.

"Ransomware gangs often destroy your backup infrastructure. So it's important to create pull-only backups or backups that cannot be deleted / overwritten."

Every rsync.net customer has ZFS snapshots available in their account that are immutable. They are read-only.

So, even if Mallory trashes your primary site and then gains access to your rsync.net credentials, the daily/weekly/monthly snapshots cannot be destroyed.

Immutable backups are often overlooked. At borgbase.com, we call this “append-only” mode and the large majority of repositories uses it. With S3 (or similar) you would add some policies to disable deletions. So it’s usually doable, but needs to be considered when setting up the backup process.

Does the append-only mode you have in borgbase fix the issues of the append-only mode in Borg itself? https://borgbackup.readthedocs.io/en/stable/usage/notes.html... Because the way it works in Borg isn't really workable in practice, it only seems to be good to check a box.

We use the same public version of Borg, so those limitations apply. It’s still quite workable. Just prune from a different machine or once a year when the repo gets too large.

We may move to reject destructive commands from append-only repos in the future, but this will break existing workflows. Discussion on this here: https://feedback.borgbase.com/posts/28/reject-delete-and-pru...

That’s all well and good for weather data, but if anything in your backup is in any way related to user behaviors or transactions, immutability is a crime.

Immutability in this context doesn't mean "kept forever", it just means the permission to hard-delete is separate from soft-delete.

GDPR requires that good data-stewards keep backups:

the controller and the processor shall implement... the ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident

https://gdpr-info.eu/art-32-gdpr/

For more details on GDPR and backups, see https://www.backup-systems.co.uk/blog/6-gdpr-implications-on...

It would be inappropriate, for instance, to store user data on the blockchain.

It also requires that user data can be hard-deleted on command. The ability to wipe data from backups needs to be kept close at hand.

> backups that cannot be deleted / overwritten

That gets complex if your database contains PII. If a user asks for their account to be deleted...

It's not "cannot be overwritten ever". It's more "you can overwrite it using credentials stored in a safe location and never from the production environment".

Someone will always have permissions to remove or change the entries, just not easily and not in an automated daily process way.

Would it be sufficient that the delete accounts script be managed and merged at restoration time?

The neatest way I heard is to store PII in an encrypted form and then delete encryption keys when a user requests deletion. That requires the keys to be in a different backup pipeline though.

You still have their data and anyone who bypasses the restoration script can get it so no.

This is not true. The prevailing opinion is that backups do not need to be modified and that it’s enough to delete at restore time (and to of course not process the data again).

Reply on news.ycombinator.com