Comment by sillysaurusx | Hacker News Reader

Comment by sillysaurusx | Hacker News Reader

One thing I've always wondered: How do you prevent ransomware from ruining your backups, too?

Lots of ransomware tends to be "delayed" – it can't encrypt everything instantly, so it encrypts a little bit each minute. During those minutes, isn't it conceivable that the now-encrypted files replace your old backups?

I suppose this isn't really a "backup" but rather a "mirroring strategy." But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy?

The other question I have is related to that first point: since ransomware can't encrypt everything all at once, how is it possible for your system to continue to function? As you can tell, I'm a ransomware noob, but it's quite interesting to me from an engineering standpoint. Does the system get into a "half encrypted" state where, if you rebooted it, it would fail to boot at all? Or does ransomware targeted at businesses tend to be more of a surgical strike, where it targets and wipes out specific datastores before anyone notices?

(It's the "before anyone notices" part that I'm especially curious about. Isn't there some kind of alarm that could be raised more or less instantly, because something detects unexpected binary blobs being created on your disk?)

Other replies have backed into this, but the best solution is to

1. Use a COW (copy-on-write) filesystem like btrfs or ZFS

2. Set up snapshots to be taken periodically (hourly/daily) and sent to a different host or volume.

3. Monitor disk usage: if you get hit by a cryptolocker, your disk usage will approximately double as it rewrites all your files.

4. Manually backup snapshots or the full volume to offline storage every N days/weeks/months.

In case you missed it, I wrote this up a while back: https://photostructure.com/faq/how-do-i-safely-store-files/

TL;DR: Lots of copies keeps stuff safe!

5. Don’t use (only) direct disk access to backups

I think all of that is lost if the cryptolocker just formats any volumes named Backup.

The copy-on-write can’t just be enforced by the filesystem. If this computer can permanently delete content on the backup system, then so can the locker.

Another important step - make the offiste backup 'pull' based - so the credentials to access the data already there do not exist on the system being backed up.

Yes!

My homelab, admittedly low complexity, uses a NAS device that powers on at a scheduled time, mounts the shares it needs to, runs the 'pull' backup, unmounts, and powers itself off.

The intention being that, in the event of intrusion, it's presence and accessibility are limited to the window of time in which it's performing the backup.

Additionally to that, a rotating set of removable HDDs as backups of backups that also get spread off-site amongst family members houses.

I really should go into offering backup solutions to local small business...

What's the point of COW? There are 0 tools that restore the "orignals" of the copies on write.

In this context COW typically comes with cheap snapshots. And restoring from snapshots is trivial.

2. Always pull the snapshots from another host with tools like syncoid. This host must be inaccessible from network so it can’t be infected.

> 2. Set up snapshots to be taken periodically (hourly/daily) and sent to a different host or volume.

Does btrfs or ZFS have a way to pull snapshots in a way that they are encrypted on the client side ?

Ideally you could hire a third party to pull these backups from you, have them warn you when the process fails or doubles in size (the data is being encrypted) and still be able to prove that there's no way they can access the data. And then the private master key(s) go into a safe.

> I suppose this isn't really a "backup" but rather a "mirroring strategy."

Correct.

> But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy?

Yes. Make the backup system for that big slow-changing data a moderate amount bigger than the primary data store, and then you can have months of retention at low cost.

If too much data changes at once then it should go read-only and send out a barrage of alerts.

You just use cold backups.

For home, I have two USB disks I use for backups and I alternate which I use. Neither is plugged in at the same time. At least one is always "cold".

For larger scale, you can do the same thing with tape. One tape backup goes off-site (perhaps) or at least cold.

The cost isn't that high. A USB spinning disk may cost a third to a 5th that of your SSD hard drive. And you can get hard drives up to 18TB now. But even a portable 2TB USB-powered 2.5in external hard drive is only $60, so this is a cheap and robust strategy.

That mitigates the risk, but it relies on the assumption that you'll notice when files get encrypted. It's not a guarantee, malwares can hide themselves long enough for you to plug both disks before you notice.

True, but I don’t rotate them every day. Maybe once a week or month. Unlikely to not notice by that time.

With tape backup, you might keep a tape in cold storage for years.

> For home, I have two USB disks I use for backups and I alternate which I use. Neither is plugged in at the same time. At least one is always "cold".

Why not have one of those drives off site, and rotate every so often? Carry the drive with you when you swap, so that the original and all backups are not in the same place at the same time.

I have three external drives. Originally I planned to keep two offsite, but I don't have an offsite office anymore.

Your general backup strategy should follow something like doing full/incremental and/or snapshot based backups. So in the case of your media, if you do a daily snapshot then your daily backups would be very small. And if the media doesn't change that often you can keep weekly copies for several weeks, monthly copies for many months, and several years of yearly snapshots.

The other strategy is with tape rotation. You need to have about 30x the tape storage as you have online storage, so you can keep 7 yearly backups, 12 monthly, 6 weekly (all full backups), and 7 - 14 daily incremental backups.

I use Amazon S3 buckets, with the web server having an IAM user key with very limited access that only allows writes.

No reads/listing.

So if the web server gets hacked, the hacker can only write to the bucket, but has no way to know what is already there or access anything in it.

Make sure you have versioning turned on as well. Even if attackers can figure out your naming conventions and overwrite, you just go back to the first version and everything is good.

Would be better if there were a “create/write to new file” permission which doesn’t allow overwriting existing files.

> I suppose this isn't really a "backup" but rather a "mirroring strategy." But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy?

Saving diffs/snapshots will solve the issue. As long as the file doesn't change the cost is almost 0.

If you decided to use incremental backups to mitigate against this, what are your favourite tools or providers? Backblaze with duplicati? Duplicity with s3? Rsync and rclone without it being incremental?

Granted I'm a complete amateur here, but still I wonder if my approach is helpful. My home backup drives are running on Raspberry Pi, so I have control over what other software runs on them. I've been writing Python programs that run on the Pi and monitor for changes. My hypothesis is that if I do the right analysis, I will notice changes to the files unless the ransomware is capable of actually infecting the software running on the Pi itself.

I believe that I would detect unexpected binary blobs. Of course this depends on me writing the programs correctly, and a lot of other assumptions, but it might suggest a way to protect backups.

My backup "drive" is anything but passive.

The change you detect would be your files being encrypted, this seems pointless unless you keep automatically rollback changes that don't seem okay to you (which means you need backups for your backups to be compared to...).

Indeed, this assumes the correct files are somewhere but not accessible to the family computers.

Linux executables can be modified while still running, this is what differentiates windows updates, many requiring rebooting, from Linux updates. Once rebooted, then you realize your executables are tainted.

Reply on news.ycombinator.com