Don’t Wanna Pay Ransom Gangs? Test Your Backups

No it isn’t easy, but it’s also not an impossible task.

Is it easier to test after the ransomware attack?

This modern attitude of compartmentalization, extremely localized expertise, and outsourcing everything to the cloud is going to be our downfall. Stop hiring muggles!

However for a company of our size, it's not really possible. I was talking to my lead about this last year and we have about 50TB of our entire system I believe stored in our databases. All he said was "we have one, but hopefully I'm retired before needing to find out how to restore from it."

Maybe we should get into the habit of going all Shiva/Brahma on that environment every week or month. Burn it to the ground, and recreate it with an automated process. Sort of like a meta CI test.

You can't fix already broken processes. VMware solved this 20 years ago. It is pretty simple to restore VMs on different systems, you don't need to worry about the ball of mud when you can duplicate it.

It's difficult to do, yes. But it's difficult in a "dig a tunnel through that mountain" way, not in a "solve this mysterious math problem" way. It certainly could be done. It would just take time and money (money including hiring people).

People constantly point to the difficulty of backups and the difficulty of hardware level separations between systems. But these are merely difficult and costly. "No being hacked" and "writing always secure code" are impossible and so they won't protect from backup failure.

And yes, companies would rather spend money on other things. That's what it come down to.

I’ve worked with 9-figure turnover entities broken by this sort of thing and the first recommendation is always fire or manage out the CIO, risk/audit officer and/or CFO.

Everyone cries about having no money. What is lacking is an ability to identify and manage risk that puts the existence of the company as a going concern at risk.

Bare metal backups are a good place to start if you have a complicated system.

A lot of environments aren't as complicated, and you COULD just pull your hard drive out, put in a spare, and test how your backup works. At least, test it while airgapped to see if it comes up at all.

Keep offline, airgapped backups.

That process works well till ransomware comes in and destroys every server and client machine at once, and suddenly you've just given the IT department multiple years worth of work to all do at an emergency pace.

This is a non-issue with Git + Ansible, even without getting into tools like Terraform. At my dayjob, i set it up so that there's an Ansible playbook that does around 200 administrative tasks for each of the servers for a particular environment - all of the configuration is within Git repositories. Changes are applied by CI, all of the process also being thoroughly documented for local development, if necessary.

Everything from installing packages, creating or removing user accounts, setting up firewall rules, setting up directories and sending the necessary configuration, systemd services for legacy stuff, container clusters, container deployments, monitoring, container registries, analytics, APM and so on are handled this way.

Noone has write access on the server itself (unless explicitly given in the playbook, or the admins) and even the servers themselves can be wiped and reinstalled with a newer OS version (mostly thanks to Docker, in which the apps reside in), plus all of the changes are auditable, since they coincide with the Git repo history.

It took me maybe 2 weeks to get that up and running, another 2 to handle all of the containerization and utility software aspects. I'm not even that knowledgeable or well paid, there's very few excuses for running setups that don't let you do things "the right way" in a somewhat easy manner, like Windows Server. That's like picking a hammer when you need to screw in a screw - it'll do the job but chances are that there will be serious drawbacks.

The primary problem you are protecting against is that backups are broken, or corrupted. A broken, as in failure to write, backup won't restore. A corrupted backup probably wouldn't restore either, even if it did it would fail the MD5 checks. No one is expecting you to destroy and repair Prod on a weekly basis.

30 years ago mainframe companies started realizing that their mainframes couldn't restart anymore - after many years of uptime all the on the fly configuration changes wouldn't be reapplied and so the whole couldn't restart. (all hardware had redundancies and backup power supplies, so any individual component could be replaced and most had been over time) So they started scheduling twice a year restarts to test that the whole could come back up. The mainframe itself is fully able to run for years without the restart, but the configuration wasn't.

The other strategy is with tape rotation. You need to have about 30x the tape storage as you have online storage, so you can keep 7 yearly backups, 12 monthly, 6 weekly (all full backups), and 7 - 14 daily incremental backups.

Correct.

> But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy?

Yes. Make the backup system for that big slow-changing data a moderate amount bigger than the primary data store, and then you can have months of retention at low cost.

If too much data changes at once then it should go read-only and send out a barrage of alerts.

1. Use a COW (copy-on-write) filesystem like btrfs or ZFS

2. Set up snapshots to be taken periodically (hourly/daily) and sent to a different host or volume.

3. Monitor disk usage: if you get hit by a cryptolocker, your disk usage will approximately double as it rewrites all your files.

4. Manually backup snapshots or the full volume to offline storage every N days/weeks/months.

In case you missed it, I wrote this up a while back: https://photostructure.com/faq/how-do-i-safely-store-files/

TL;DR: Lots of copies keeps stuff safe!

No reads/listing.

So if the web server gets hacked, the hacker can only write to the bucket, but has no way to know what is already there or access anything in it.

Saving diffs/snapshots will solve the issue. As long as the file doesn't change the cost is almost 0.

I believe that I would detect unexpected binary blobs. Of course this depends on me writing the programs correctly, and a lot of other assumptions, but it might suggest a way to protect backups.

My backup "drive" is anything but passive.

For home, I have two USB disks I use for backups and I alternate which I use. Neither is plugged in at the same time. At least one is always "cold".

For larger scale, you can do the same thing with tape. One tape backup goes off-site (perhaps) or at least cold.

The cost isn't that high. A USB spinning disk may cost a third to a 5th that of your SSD hard drive. And you can get hard drives up to 18TB now. But even a portable 2TB USB-powered 2.5in external hard drive is only $60, so this is a cheap and robust strategy.

A note for anyone else in a similar situation - a good team doesn't ridicule someone for questions like these. A responsible leader should have cited a time in the past that they did a restore or a spot check, and no one should have laughed. The laughter sounds like masked fear or embarrassment.

This goes for any team. "How do we know this function of our job does what we think it does?" You should have an answer. Now, I've only worked in R&D software and not in IT. But IMO IT teams should work the same way in this regard.

Modern tape drives (like LTO) will at least do a read after write so you should never end up with blank tapes after a backup. But still no excuse not to do restore tests.

And make sure you're not storing your backup decryption key in the same backups that are encrypted with that key. Likewise, make sure you're doing restore tests on a "cold" system that doesn't already have that decryption key (or other magic settings) loaded, otherwise you may find out in a disaster that your decryption key is inaccessible.

Also the article doesn't seem to consider the fact that some hackers are now threatening release, not just destruction. Embarrassing emails, source code, and trade secrets. Backups won't help at all.

The whole post should be printed out and pinned to the Kanban/Scrum/whatever board of every infrastructure/DevOps team, but this sentence in particular. This property of Azure (and I imagine every other cloud provider) was one of the nastier fights we had with the guys who run the on-prem firewalls.

In order to not lose data, you can't have any writes between the time when the backup was taken and the present, or you need code which reconciles additional state and adds it onto the backup before switching over.

Normally, backup restoration is done during a maintenance window where the site is disabled so no writes can happen, and then usually a window of writes are lost anyway (i.e. 'last X hours, since the backup was taken')

For your use-case, do you just have very few writes? Do you lose writes? Do you have some other clever strategy to deal with it?

I like this approach, although risky if you mean you routinely replace the production db.

My preferred setup is to automate restores into a pre-prod environment, apply data masking and run tests there. It's not a replacement for full DR exercises, but at least it automates the backup verification process as part of your build system.

The stack replaces the previous one or the backup replaces the previous one? While having a single backup is a good start, you might want to consider keeping several backups so you can restore from, say, a data entry error that you discover two months after it happens.

It won’t stop a root account compromise unless you’ve got a multi-account setup going (as they could edit the bucket policy).

But if you’ve not got any monitoring, they could also just remove the lock on the objects without you noticing and wait for the timeout themselves.

Most of the remaining suggestions aren't relevant to ransomware even if they're otherwise mostly fine recommendations. 2FA won't stop ransomware, or data destruction. Redeploying code and reinstalling active directory doesn't restore customer or user databases. Paper files are not accessible when they're needed, are easily damaged or misplaced, and cost a lot to store and index (if you're referring to keeping them as backups then yes you're making a form of the argument of the post just in a very expensive and inconvenient way). Read-only S3 copies are almost certainly falling in the realm of backups... but is also a relatively expensive way to do it for most organizations larger than a start-up.

Offline and offsite backups are the cheapest, most effective tool for keeping copies of your data in your companies possession due to unforeseen circumstances and they protect against a huge number of potential events beyond just ransomware. It's negligent IMO for executive officers of a company to not have invested in a solid, tested, and comprehensive backup solution.

I work at a big company that does this, but with 6 months. While I understand why they would do that, often some knowledge is lost with these emails. And usually I don't know what's knowledge and what's junk before I need it.

On the other hand, it's a good way to make sure that your processes are written somewhere and people don't rely too much on email archiving. Sadly, that's something I didn't realize until it was too late.

And would you trust you data after criminals had access to it?

1. Figure out what's valuable and what needs backed up. Some will say "back up everything", but then you don't know what you need to restore and how to restore it! You need to know what's valuable. Is that source code? A complex application environment? Files in S3 storage? User generated files stored locally on a server? Data in a database?

2. Figure out where it's stored. Not always straightforward, but again necessary. Otherwise you won't know how to back it up, or recover it. Remember the cloud isn't a backup - when someone steals your production S3 keys, absent some careful configuration, they can dump and delete all your bucket contents...

Understand what you need to backup and how you'll recover it. If you are dealing with Windows based environments, consider the practicalities of getting these up again - windows isn't as nice as Linux (which can boot on any hardware within reason, modulo some bootloader settings in a few complex configurations)

3. Figure out how you're going to protect the data you back up. Now you've identified the crown jewels, don't just store them in plaintext and unencrypted on your laptop (!) If you're going to use encrypted backups, understand where the keys are stored and how they're used. More importantly, understand how you'll have the keys to decrypt the backups after a disaster.

4. Figure out how you'll stop a someone who gets in from deleting or accessing your backups. Assume they'll be attacked. Understand what the backup solutions offer in terms of security. Can you use asymmetric crypto so only public keys get stored on your servers? Can you use protection on an s3 storage bucket to prevent deletion, even with the access key? Etc.

5. Test doing backups. Then practice restoring to a non production environment to see what you missed. Document it all!

6. Set up monitoring so you know when something goes wrong with your backups. Gather and expose enough information to yourself that you know what's going on! This could be as simple as an automated email to yourself every morning, telling you the duration of the backup job, and the amount of data written, and amount changed since last time. But you need to monitor this. Don't just go for a failure alarm (have one of those, and make it fail-safe so it alerts if the backup didn't succeed, even if the backup script didn't run at all!) - also notify yourself on success so you are aware of the backups and what goes on.

7. Ensure you have enough copies. 1 isn't enough - what if the payment card on your storage account expires, or you get locked out? Or the company goes under? People normally say 3 backups in 2 locations, at least 1 off-premises.

8. Return to 1, because by now something has changed and you need to add more new things to the backup system.

There's all sorts of other things to consider like data consistency and whether operations on your systems are atomic (what happens if the backup snapshot is taken mid-operation in your app?) that you'll want to think about too.

Espionage, and blackmail are both crimes that have been around for a very long time and have international treaties around them. That doesn't stop it from happening, but when someone is caught it's much more likely they're going to be extradited and prosecuted than for a crippling attack against a computer system.

Either way, if an attacker is threatening to release your internal confidential data, would you rather have a copy of that data as well or not? Your course of action is also the same in either case, notify the FBI if you're in the US.

That's the part I've always had trouble gettin' out of most "management" types. They hire you for your expertise, and then undermine it at every opportunity to "save money" or to exert their "authoritah".

Assuming you do get some capable security people in, they're part of a "cost centre" - most organisations still see IT as a cost to the business they'd love to eradicate, rather than as a key enabler that allows the organisation to exist. I had hoped covid would cause a shift in mindset as companies realise the enabling effect their IT teams had, but old habits die hard, and it looks good to recharge IT to lower your perceived overheads of doing business by billing other departments internally for IT. That leads to cost cutting and the other issues you pointed out.

Even then, on your final point about listening to them, I share your frustration. Again the common complaint I get is that the security people don't speak the same language, so neither understands the other, and the conversation ends. The security team expect the suits to know why it's bad that the office printer is 15 years old; the suit feels that's prudent cost cutting and assumes it must be fine because it came from a reputable brand.

Ultimately security people need to better communicate to stakeholders that the starting point is for everything to be insecure, and that security is needed to make it secure. And left untouched, it will eventually end up insecure again, through not being patched. Unfortunately this message is just perceived to be self serving, as it's exactly the same message every other department is giving - "our team is really important, give us more money to...."

Some other thoughts in relation to your points:

- the continued insistence on flat network structures with file shares and similar is a huge issue. Same for the security posture of a Windows server in a corporate environment - it's almost entirely based around the idea of a trusted LAN. That's an outdated set of assumptions, but is very often how malware spreads. There's zero reason for workstation to workstation traffic originating from any part of the organisation, irrespective of protocol. Give Devs a separate environment without restrictions, and let IT use a secured jump environment to do their remote connections. Preventing end user devices talking to each other at all would be a good first step.

- Next up would be getting rid of large network shares that half the organisation has read+write access to. Something HTTPS based, with proper logging and 2FA would be a better starting point. Rate limit requests and monitor the logs on the rate limiter. Convince Microsoft somehow to move AD towards a zero trust architecture and run it over HTTPS like a modern service, rather than legacy protocols, or preferably move to something that doesn't require multiple gigabytes of other likely vulnerable services running (DNS, print spooler, file shares, etc) just to give you AAA.

- Security isn't something anyone wants to pay for until it's too late. Businesses often see cyber as another risk on the risk register, and they try to treat the risk through insurance. In the longer term this won't work, because it is becoming a near certainty that the average organisation will be compromised. Insurers don't like to cover for certainties(!) If businesses just see cyber as a financial risk that happens once in a blue moon, expect them to extrapolate the costs per breach and set your budget based on the cost of a breach split across 5 or 10 years. Defenders' dilemma.

- Snake oil security sales pitches very effectively target the MBA suits directly and sell them over hyped claims. You'll then end up pressured to use your finite security budget on their ineffective snake oil, which doesn't actually achieve anything much (and likely slows down systems). This leaves you without budget to develop internal bespoke tools for network monitoring. It's always entertaining to see how many companies can tell if their users iPhones were affected by (for example) NSO Group - can they actually check DNS logs for presence of IOC domain resolution, or do they lack even that level of visibility? But the basics aren't exciting, and the big vendors send well-heeled sales people in with dark backgrounded slide decks to inspire MBA-laden confidence in their snake oil.

“That is still somewhat rare,” Wosar said. “It does happen but it’s more the exception than the rule.

It's something that very high-value targets will eventually have to worry about. But figuring out how to corrupt the backups without disrupting the production systems (and being discovered) is not likely to ever be a fully automatic process for the randomware goons. They'll have to invest serious time in understanding how the victim's systems work. For high-value targets, that will happen.

I just walked through a telecom colo last week and saw a weird new box in somebody else's cage I'd never seen before. Later, I did a web search for the brand name. Apparently this company sells a network filer that keeps local daily/monthly/annual backups and doesn't let you delete them. Probably a good idea for an organization that just wants to throw money at the problem. Of course you can build this yourself with "btrfs sub snap" except for the part about not letting yourself (even as root) delete them.

Is this something companies want to gamble their future on?

Human mistakes, hardware errors, and the increase of ransomware. That, and many companies relies on their IT infrastructure for their existence. I think backups are well worth it.

Backing up your system for disaster recovery is something that has been known as expected for ages. If you’re too stingy or dumb to realize and do such processes (especially years after the first ransomware attacks) your business definitely deserves to die a horrible death. Only hope is innocent people don’t get affected by it. But they probably will.

Two from the '89 quake.

1) The smart thinking operator who equipped their machines with UPSes to keep them up through a power failure. The quake cut the power, the UPSes kicked in, the systems stayed up, the drives kept spinning, the quake kept going and buried the drive heads in to the platters.

2) System crashed, backup tapes were kept in the basement. Quake set off the sprinkler system in the storage room, soaking the tapes with rusty water.

"I'm sorry, did I break your concentration? I didn't mean to do that. Please, continue, you were saying something about best intentions." - Jules