Hacker News Reader

Comment by osinix | Hacker News Reader

The backups got wiped together with the systems, so they were reachable from same network. A backup the attacker can reach is not a backup. Good they had an offline copy, but a system this important should have that as regular schedule, not depend on luck.

View on HN

    > backup the attacker can reach is not a backup

you can have append-only backup systems.

That was my thought exactly on reading that line: that is not a backup. (Ok, the word isn't strictly defined, but you know what I mean.) They have said there's a "real" (offline) backup as well, luckily, but that just reinforces that the "pretend" backup was irrelevant and wasn't even worth mentioning in the writeup.

Any guidelines on how to back up such that the attacker cannot reach (when the hacker otherwise had some valid credentials)?

This is how we implemented this at our company:

- We have 2 sources of data that we must backup to continue existing as a business; our postgres and binary files in S3. Everything else is derivable (elasticsearch, so on).

- For postgres, we use barman. With the help of opus/fable, you can get a streaming replication backup working in no time. We have one into another server in the same datacenter (we use baremetal) and another one in another server in a different datacenter.

- We then have a last resort barman backup with bi-weekly base backups + WAL streaming to S3 (both the base backup and WALs). It sends these backups + wal segments into an specific S3 bucket that has object lock in compliance mode. This is a feature from AWS S3 that even the most privileged account credentials (super admin) can't turn off nor delete the files before the object lock, which is 10 days in our case. Object lock compliance mode can only be extended, never shortened.

- For S3, we store them into another versioned bucket, with lifecycle rules to also expire non current versions (== deleted objects) after 10 days. No point in object lock compliance here because it would only protect objects for the most recent 10 days, and you gain nothing. What we do instead: the app servers only have access to these bucket tru an IAM credential that can't delete old versions (so deleted objects have to expire manually via the lifecycle rule) AND this IAM credentials also can't change the object policy.

IMHO, this protects us enough so that even in the worst case scenario (ransomware) we have 10 days to sort everything out and recover our AWS access.

And yes, we test the S3 barman restoration and it works fine. Data loss is at max 5 minutes due to the archive_timeout=300s on the primary.

For the streaming replications in the two servers I mentioned, it's less <1ms, but those wouldn't protect us much in the case of the ransomware - even tough we use tailscale and one compromised server can't ssh into the other.

You need something that's not online or at least not with credentials and devices the Devs and operations people use.

The worst case is the company itself denying you access. More likely to happen with Google.

So if a hacker infiltrated your system and silently fed plausable but wrong data into your databases for 10 days, your whole system would be screwed?

My friend, our users would flood our support within 30 seconds of the first blip of this happening.

I'm curious: what's _your_ defense against this?

i have daily backup instances to cold standby qemu-kvm VMs with snapshots of qcow2 storage which are kept for 10 days and i skip deleting every weekly Monday backup until space becomes a problem (usually months). sometimes the qcow2 storage backend becomes too slow with so many snapshots and that's when i qemu-img convert it which basically fixes it but removes the snapshots, so i keep both of them for a while.

You can use old school way. You can have a tape backup, ssd, or a device that is not connected to the network constantly. Copy there and unplug the device.

Reply on news.ycombinator.com