How do you create a backup server that is reachable by production servers (so that they can back up to it) without then being vulnerable to the same kind of ransomware attacks that infect the production servers? You can't exactly make them read-only, or else they can't accept the "legitimate" writes that might occur during the normal backup process.
Production has no access to backup.
Backup has read only access to production.
Backup writes are append and not overwrites.
Deletes/archival are governed by a retention process.
It's nice to have a restore step in there too, so one has both a validation that the backup is usable, and gives a "playground" where one can have a safe day old environment for testing/training/whatnot against production data.
That is essential. If you do not regularly test restoring, you do not know if you have backups. I normally overstate this just a little for effect, "If you don't test your backups you don't have backups", but that isn't precisely true.
And yes, it frequently makes sense to base test/fixture data and whatnot on backup data flows, but unless you don't deal with any PII, you're hopefully not using that data without a sanitization step.
To anyone viewing my answer as a guide: a backup by itself is not a continuity plan. There are other important areas such as setting recovery metrics (RPO/RTO) and testing restores against those metrics. The world is rife with stories of unreadable backups and/or restorations that took days instead of hours.
1. Gain access to change the code on the front-end web servers (usually PHP)
2. Change the database access layer to transparently encrypt data being written to the database, and decrypt data being read from the database. The key would be loaded into memory by curl'ing an attacker-controlled website at startup.
3. Wait 30 days
4. Notify the company that they're compromised, turn off the attacker-controlled key service, and restart the web front end
Now step (3) ensures that most data in the database has been re-written, and if your backups are dumps of the production database, you now have a month of encrypted backups that you can't read... If you're lucky, you may have a month-old backup to restore from; if you're unlucky, you rotate every 30 days.
Implementing zero downtime transparent encryption on any moderately complex codebase above the database layer sounds like a big engineering challenge. Even moreso if you're trying to do it without the sysadmin noticing. Even moreso if there is no persistence for the encryption key so every user request ends up having to fire a request to the attackers server - that'll kill performance. I doubt many attackers would attempt it.
Read through any modern multi stage remote exploit. The amount of engineering required to pull these stunts off - reliably no less - is truly staggering, and yet attackers continue to innovate and develop new techniques and even more complex exploits.
And you overestimate the competency of the defenders: sysadmin? What small business has a dedicated sysadmin? It’s not like we are talking about running a global hosted chat service like slack here. (Oh wait...)
BTW Persistence for the key in memory is trivial: create a shared memory segment and store the key there, with a ttl.
Was it a site in maintenance mode, where no one was working on anymore?
Oh, and there must not be a DBA at the company either, since absolutely nobody ran a SQL or analytics query in an entire month and noticed that some of the rows were full of unreadable garbage..
This story must be about someone's single server, single dev LAMP blog from ten years ago if it ever actually happened.
Yes, we are talking about the long tail of little LAMP servers handling small business needs with some custom code written by a contracting shop. What’s a devops? DBA? Once the site is developed and working, there’s little reason to pay to change it.
Edit : thinking about it, this story doesn’t add up. Doesn’t this mean the client must have a copy of the decryption key, meaning any cached client would render the ransom demand worthless? It’s just also so much easier, if someone has that level of access, to make a copy of the database, encrypt it, then blow away the old one. Doing a silent deploy of client code with no one noticing seems way harder.
Honestly, it used to be Eastern Europe, until we started realizing they've got some serious programming talent and contracted some work out. Now it's less bad, though still a bit sketchy. Of course, this doesn't help nation-state attacks.
Where should I send the resume?
Where it breaks down is assuming only the web server that is compromised has access to the database.
Of course, it's also possible it's not, and I wonder if that's more likely. Every industry has such stories; maybe this is the white whale of forensics.
The network is segregated to limit impact if they do get hacked.
To keep everyone familiar, Data Recovery processes are practiced regularly.
...
This gives me the heebie-jeebies.
It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing.
I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.
10TB is not even the largest single HDD money can buy... and I wouldn't use a single drive anyways. What I'd do... There's the new Silverstone CS381 chsasis with is <30L (so it easily fits anywhere) and houses eight 3.5" hot swap disks plus 4 2.5" for boot and a micro ATX PC... Looking at https://diskprices.com/ I'd go with the Seagate 8TB drives at 120 bucks by shucking them from external and then you need a cheap uATX board and an LSI-9207-8i card.
That case does look very nice for a storage box though.
You just go back to the last good version.
Cronjob to an (S)FTP server and an upload script trigger to chown/chmod all incoming files making the whole thing WORM (Write Once Read Many).
Once its submitted the same user account can't alter it. Even if the malware is clever and scans for .netrc and .id_rsa and manages to create its own connection to the backup server it doesn't have access to anything anyway.
- I have a UnRaid machine, and a backup machine. The backup machine is a small itx board, and has a single HDD attached.
- A NodeRED instance has a so-called "Flow" on the UnRaid machine that is waking up the backup machine every 7 days.
- Thanks to anachron, with a 10 minute delay, rsnapshot connects to the UnRaid machine, pulls the data, and then issues a shutdown to the backup machine.
This setup let me sleep pretty well.
My home computers push backups regularly throughout the day, and every day I create snapshots of each volume (how long to keep the snapshots is another question). This snapshot can only be accessed or managed on the NAS itself.
This effectively creates an append-only backup NAS thanks to the periodic snapshots.
https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.htm...
As usually with security, the principles of least privs and segregating as much as possible are important.
Of course IANAL so idk how this jives with various EU laws.
EC2 -> S3 bucket with only write access and versioning enabled. EC2 -> EFS and it's a rotating set of 7 with 7 different security groups that rotate.