Hacker News Reader

Comment by mysterypie | Hacker News Reader

Could they have avoided the ransom by having daily (or hourly) backups to non-rewritable (write once) media? So the malware won’t encrypt the backups, obviously.

I think that the last day’s work (or last hour’s work) will be lost or will require a lot of manual fixing regardless of whether they pay the ransom. If they pay, they’ll still have to fix partial database transactions, corrupt files, etc., for the attack date. If they don’t pay, they can recover from earlier good backups and reconstruct that one day’s worth. My reasoning is that the attack date’s data is going to be corrupt and untrustworthy in either case, and it’ll be equal work either way. (Or at least it’ll be less than $600,000 of work to fix that one day.)

I imagine that they either weren’t doing backups at all, or their backups were directly accessible and writable by the malware.

View on HN

I work for the city of Skanderborg in Denmark and we do a few things to avoid it. One is to monitor our entire storage for malicious code and automatically isolate suspicious activity. Another is to do frequent backups of everything on our network shares and one drive for business which is where most employee data that doesn’t belong in a specific system lives. Our servers, database clusters and vital systems are all isolated from the employee network and also frequently backed up. When we get hit, it’s usually employees reading private email and we’re typically able to isolate the ransomware before it spreads from that specific employees network share. Once we kill it, we restore files to the most recent backup and roll their machine.

It’s worked well so far, but we do have an IT crew that would make most places jealous and IT is an area that is notoriously undervalued in the public sector.

See this seems to be a really fun and challenging part of IT.. the part that I don't want to have to deal with is the helpdesk, guiding a 60 year old lady to login onto her system or configure her company iPhone.

Maybe there is a job where you do only the former, I just haven't found it yet.

Network/infrastructure guys don't seem to me to do helpdesk very often?

It depends on the size of the company.

What do you think happens when a core core router breaks? The level one guy fixes it?

There are a few Palo Alto and Cisco guys near me. They definitely aren’t answering the help desk phone but they are level 3 support. Same with my development team.

Not to mention large orgs have NOCs dedicated to this. The last company I was at at a couple CCAs in the NOC. They were smart but an odd breed.

Our IT is two things, one is helpdesk the other is “tech”. Tech handles servers, networking, security and Azure and only really offer support for developers like me.

Same here

These will be separate jobs in any large enough IT org.

If you are unwilling to get your hands dirty interacting with the people who use your network you'll be unaware of how your decisions affect their day to day lives and consequently they'll probably end up hating you, which will not do you any favors in the long run. It is important to remember that everything you do is ultimately in service to helping them do their jobs.

But guiding that 60 year old lady through configuring her company phone is how you discover that your process for configuring company phones is broken. When the people who design a system never interact with the people who use the system, that is how you end up with shitty unusable IT systems.

I imagine that they either weren’t doing backups at all, or their backups were directly accessible and writable by the malware.

Or they weren't able to detect exactly when the malware was deployed so they didn't know how far back the data corruption went, meaning they couldn't trust the backups even if they looked OK. One of the problems you face after any attack is trusting the system again. Verifying everything is correct is a very hard problem.

I think there is alot more going on here than just some encrypted files.

The interim IT manager had to setup a new website for the city w/ new email addresses?

They lost control of DNS and registration of their domain, it seems.

SCADA systems for water pumps were inoperable.

Online payments no longer functioned.

This seems like a very premeditated attack to me.

Knowing all the "flat" networks I've seen that all sounds par for the course. Segmentation is still exotic (and met with either disdain or blank stares by software vendors). The attack damaged all those systems because, very likely, they were all accessible to each other and under one locus of control (probably a single Active Directory domain).

SCADA functioning is not dependent on AD.

The AD server should be able to be destroyed without preventing the water supply from functioning.

I know of three small municipalities in my area (smaller than this town), and the utilities are not part of the flat network.

More than likely the SCADA systems themselves were fine, but the PCs for managing those systems were AD joined and people couldn't login to actually run the management software.

Its very common in custom hardware setups to have a standing system that interfaces with the physical hardware and PLC and then the user friendly software for instructing that controller on what you want to be on a PC talking to it over serial or the network. Obviously if the computer is inaccessible you can't adjust settings, but the system continues to run fine.

Right, but the story states otherwise.

A very good solution here is ZFS. Office file server runs ZFS and makes regular ZFS snapshots (hourly, or more often). Snapshots are immutable so even if the employee machines get infected, nothing can be lost (except the last N minutes since the last snapshot). Obviously, still have to do backups (off of those snapshots).

> Snapshots are immutable so even if the employee machines get infected

Not really. You can easily extract data and overwrite the disk surface.

Not from the machines which typically get infected, which are the employee desktops accessing the file server.

If the malware manages to get itself running on the file server itself (and with root privileges), then sure. That's not a common case though.

What if the malware puts itself between the hardware and the filesystem?

I absolutely love ZFS, but I think it's only a piece of the puzzle necessary for mitigating this threat. In terms of data integrity, I sure hope no system running ZFS would have it's snapshots also corrupted, but if that did happen, then hopefully those ZFS snapshots would have been backed up to some offline media like tape or optical disk. Example: http://girlyngeek.blogspot.com/2014/02/zfs-on-read-only-devi...

Pull instead of push backups are one way of trying to mitigate this. You dont allow clients to start backups. Your backup server does instead, "pulling" backups at standard times.

As long as you harden it from crypto, there is no way for malware on client machines to force an overwrite of current backups.

Very clever.

If their backups are being placed somewhere like S3 they should have a stand alone server that simply copies daily backups from the S3 bucket the network has access to, over to another S3 bucket that only the backup managing app has access to.

I expect they are allowing their backup app to have read write access to manage cleaning up removing backups but just giving it write only access from the network would work too. And using a standalone server/app to manage the backups.

I'd still copy over a backup to a stand alone bucket not accessible by the network for something this critical.

The malware could wait for a pull, then feed the server false data.

Then you have a backup server that's privileged to access data from clients, which makes the backup server an attractive target.

Then you reduce the attack surface on this one special-purpose machine, which is much easier than doing the same on all your employees' desktops.

Most of these places have their backups on directory joined machines with storage that is badly configured. Backup systems should never be joined to the domain specifically for this reason--but it makes configuration harder so most places skip it.

We use frequent ZFS snapshots (every 15 minutes for 24 hours) then getting coarser the future out you go. Came in handy when the Dean got hit with ransomware.

or use Linux instead of Windows..

Once Linux is juicy enough it will be the new target.

Linux is way harder to target, as any software vendor will readily tell you, so it would need to be significantly juicier for worthwhile return on investment.

People have been saying exactly this for 20 years now, and it still hasn't happened: organizations still entrust Windows with all their sensitive data and critical services. The fact is, if these victims had been on Linux instead of Windows, they would have been mostly immune to attacks, certainly to these ransomware attacks that only target Windows.

Remember, when you're hiking and you're attacked by a bear, you don't need to outrun the bear. You only need to outrun the slowest hiker in your group.

Reply on news.ycombinator.com