Hacker News Reader

Comment by schoolornot | Hacker News Reader

CISO: https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-br...

I can't think of any reason not to use a cloud hosted service for backup today. OneDrive, Dropbox, and Google Drive all sign BAAs and give you versioning amongst a million other security features. AWS even has offerings that let you take periodic snapshots of on-premise volumes.

Point in time recoveries for the entire account would be nice add too but not having to fork over a million dollars in exchange for a few clicks sounds like a bargain. Hell we were setting up write-only S3 buckets for critical data stores 5+ years ago.

Gross incompetence, I'd fire everyone. A district-wide outage for a week, fine. A 1.4 million dollar check to get data that should have been archived somewhere GTFO.

View on HN

These sound like servers used by researchers. I've worked with higher education research computing and you might be surprised at what you would find.

Researchers may be generating or churning through countless TB of intermediary data, scratch files, etc. Often, the people who actually run the it infrastructure for researchers are... grad students. Sometimes they have grants for hardware and tight budgets, and paying anything for backups isn't part of it. Sometimes, if you're lucky, the it department will be aware of the work and allowed to help.

Now maybe that's not the case here, and there really is one department responsible, and that department decided against spending money on backups. Well maybe they were told by the provost or the dean of whoever that they couldn't afford to back up everything, so they should just stick to file servers. Maybe the boxes compromised here are compute only, and all code and valuable artifacts are expected to be stored safely somewhere else. And maybe the researchers heard this and understood it when they agreed to use the system. But maybe the new grad student didn't get the memo and developed his model in vim on the compute node.

The point is, academic computing is kind of the wild west. Weird fiefdoms and weird restrictions, budgetary and otherwise. It's tough to guess which of these scenarios played out from the outside and we really can't know whether the CISO or CTO, or even anybody working for them, dropped the ball.

I'm so glad I'm not the only one who has experienced this. Only thing I'd add is the lack of understanding about how hard backups are at some of the scales of data in research. Trying to explain to researchers who run the departments that doing a full backup on 35TB+ of data on all budget drives is going to take weeks was something I could never get across, until they lost all their data.

And no, snapshots weren't an option on a legacy clustered filesystem that they wouldn't migrate from.

But that is only like 3x 12TB USB drives.. </sarcasm>

> But maybe the new grad student didn't get the memo and developed his model in vim on the compute node.

Was trying to think through scenario's like that as well, but they don't really make sense.

If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it".

But mgmt decided that work was worth paying US$1.1M+ for, and copping the reputation damage.

So, it's a weird mix of potential scenarios to actually get that happening. ;)

Maybe someone internal was actually responsible for that malware and is going for some kind of weird payday?

> If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it".

Thinking of them as "staff" is the thing. If the random grad student is doing the work the pi really needs done, and deadlines are approaching, then other people's reputations are suddenly on the line. And maybe there's a deadline for a grant that would bring in millions that couldn't be met without that data. Would you pay a million now for much more grant funding?

And maybe that ransom money simply comes out of a different financial pot, but the purse strings are loosened for an important faculty member. Maybe the faculty member is hugely important but runs their own shop for vanity reasons, (which the it department really hates, by the way), but when the faculty member gets in trouble he can pull rank and raid the it department budget for the ransom.

The incentives and power structures in these organizations are complex. And the egos are huge.

Good points. :)

"I'd fire everyone." Too late. They did that 3 years ago. Fired the IT staff and outsourced to India. A great decision that saved them tons of money. /s

https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-worke...

Fire the leadership from the top down. Every one approved outsourcing the IT staff.

From that link:

> This move will save UCSF $30 million over the next 5 years.

So they're ahead?

I'm being slightly snarky here, but also earnest. If they save 30 but lose 1.14 isn't it worth it?

Depends what the total cost was for this...

The 1.14M is just the ransom, doesn't account for how much money was wasted on other aspects of recovery and total downtime...

Substantial reputational damage will have to figure in there somewhere as well. ;)

> ...Fired the IT staff and outsourced to India...

Are you asserting that the breach would not have happened if the IT operations were in-house? Could you also clarify why did you feel it necessary to qualify the location of outsourcing? If the project was outsourced to any other country besides India, this would not have happened?

> If the project was outsourced to any other country besides India, this would not have happened?

I think the emphasis is on the fact it was outsourced to another country; you can't generally check the credentials of someone on the other side of the world very easily. The University of Delhi probably has good IT staff, but “Delhi International Computer Help”? Probably not.

> you can't generally check the credentials of someone on the other side of the world very easily

If that were true, we would not have any trust in global trade. There are plenty of IT companies that have all American IT workforce and still have suffered data breaches. The issue at hand is not checking someone's credentials here. The issue is the political undertone that OP has taken.

There are reasons not to use cloud services for backups, not trusting them with your data being a major one. But then you should still be using some kind of internal backup system.

There is no excuse for not having backups.

If you don’t trust them with your data, you can encrypt it with your own keys.

Really, someone just did that for them, for the bargain price of $1.4M. I'm guessing they saved at least that much by outsourcing their entire IT department a few years ago.

This is really the best possible answer. you just did the root cause analysis for them too (you should send them an invoice).

Nothing new, in a way: cut the spending on IT, lower quality, get incidents.

Meh.

But then you lose the main advantages of using them to begin with. It becomes harder to use, it may not be able to do efficient differential backups or snapshots anymore or require you to do some complicated thing to make it work because their interface isn't meant to be used that way etc. And if it's actually important for the data to remain private, you then have to get the cryptography right, not think that your backups are "encrypted" because you used TLS, or use some snake oil encryption software which is using export ciphers or bad random number generation or anything like that.

In-house backups aren't that hard. The hardest thing is to make sure everything is getting backed up that ought to be, and actually test that it is, which is no different with cloud backups.

> But then you lose the main advantages of using them to begin with.

Poppycock. There's plenty of software that can do encrypted incrementals / differentials: Commvault, NetBackup, Veeam, tarsnap, Duplicity, ZFS snapshot send-recv,

You're asserting that these things are as easy to use as keeping all your junk on a shared Google Drive or whatever it is many companies are currently doing?

They all require you to have a level of technical competence equivalent to what would be needed to implement the backups yourself.

If you're talking about something that is worth paying US$ 1M in ransom, then paying someone a few (tens of) thousands in cost centre chargebacks upfront, and probably an ongoing annual fee, is cheap insurance if you want to go with the 'enterprise software'.

And if you want 'consumer software' that offers cloud / offsite encrypted back, then Backblaze offers it in a very clicky-clicky fashion that most folks can handle with a bit of hand-holding:

* https://help.backblaze.com/hc/en-us/articles/217664688-Can-y...

As does Arq, with a documented file format:

* https://www.arqbackup.com/arq_data_format.txt

Every cloud-capable backup system (for consumers and enterprises) does encryption nowadays, so talking about 'data access' in the cloud is a straw man.

If you really want to include NetBackup in that list, then for a real world deployment you'll generally be looking at more than US$1M+.

Assuming it's for an actual enterprise or at least large dept.

Source: Used to be a NetBackup engineer on enterprise accounts :)

I've included more and less cost options in the list to counter the very argument that you seem to be attempting to make.

How much productivity was lost from many dozens of people not being able to work because of lack of access, the IT resources that had to be extended to investigate the various options, and then the US$ 1M ransom eventually paid out.

For Want of a Nail:

* https://en.wikipedia.org/wiki/For_Want_of_a_Nail

eh, there's no free lunch.

“No, we can’t use leading cloud providers to store this data. What if someone unauthorised looks at it?”

This attitude leaves the victim paying millions in ransom. Look, Microsoft and Amazon probably have a better handle on security than your IT dept which is two people who also have to fix any issues like the WiFi not working or software not updating.

There are stories of MSPs who got their offline backup from the same provider who they bought their cloud based RMM tool.

One day an account get's exploited and suddenly the backups are deleted and the systems are all encrypted remotely.

Cloud backups are not offline backups, if you want security use tapes or remote systems which turn on manually.

A backup that can be overwritten is not a backup.

It is possible to use a cloud provider and also write an immutable backup.

> This attitude leaves the victim paying millions in ransom.

Not if they still have backups that aren't cloud backups.

> Look, Microsoft and Amazon probably have a better handle on security than your IT dept which is two people who also have to fix any issues like the WiFi not working or software not updating.

And that's part of the problem, right? Your IT department isn't too bright, so they encrypt their cloud backups with a tool that does the encryption wrong, and the smarter engineers at the cloud provider know that it's vulnerable so now they can read your data when you thought they couldn't. Or you use a weak password for the cloud backup account and then some third party breaks in and gets them.

Not really a concern with a backup tape stored in a fire safe.

There is no reason to believe clouds don't have security issues or software bugs that cause inadvertent data loss.

Just because the cloud provider is a big company doesn't mean the security is better. At the end of the day, they are a team of software engineers with all the usual people and org problems and can have usual human mistakes.

But cloud solutions are better in practice due to totally non-technical reasons. Here's how -

If UCSF can afford to pay millions in ransom, they can definitely afford to hire the right experts to do their IT systems properly.

It is likely that they hire the right people, but then the most commonly recommended security policies were considered but not implemented due to resistence from powerful non-security team members inside the organization. (Usually, the highly paid CISO is playing politics with peers to keep the job by ignoring those security experts).

If it also possible that the IT+security team is incompetent and is only good for deploying expensive but useless vendor solutions that do a lot of security theater but do very little to improve actual security. Also possible that the vendor solutions are susceptible to be misconfigured easily to become insecure.

Public cloud solutions can help overcome these above shortcomings of in-house security+IT teams in a org politics friendly manner.

> Public cloud solutions can help overcome these above shortcomings of in-house security+IT teams in a org politics friendly manner.

The problem there being that the choice of "public cloud" is subject to all the same forces, so then the vendor with the best corporate marketing team gets the business even if their security is crap.

> […] not trusting them with your data being a major one.

Which is why you encrypt the data before sending it over the wire. Just like how you'd use encryption on LTO tapes before sending them offsite.

> I can't think of any reason not to use a cloud hosted service for backup today.

Or just use a non-Windows file server that supports snapshots: NetApp, Isilon, FreeNAS, etc.

If an end-user devices gets infected, and then encrypts mapped drives under the credentials of the user in question, that won't do anything to the read-only data.

Snapshots are not backups, but for a lot of the most common situations, they offer a convenient, quick win so people can move on with their day (often in a self-service fashion by just going into a .snapshot/ directory).

Cloud solutions are great unless youre storing huge amounts of data thats being served, modified, and/or backed up. Go with JBOD's, ZFS+snapshots+rsync (or gluster even), and lots of firewalls with MFA and youll be fine (im averaging 60 to 120 TB's in my boxes raid5-7). Google drive is fine for docs. Data should stay home and be backed up elsewhere in the 'house' (still home ... just a separate basement data center or something that wont also burn down). Lets not forget this could have been an inside job that will get around a lot of security other than another humans/montiors vigilance. UCSF outsourcing all their talent is exactly what enabled this.

Reply on news.ycombinator.com