Update on IT Security Incident at UCSF | Hacker News Reader

Update on IT Security Incident at UCSF

ucsf.edu

169 points1024core146 comments

CISO: https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-br...

I can't think of any reason not to use a cloud hosted service for backup today. OneDrive, Dropbox, and Google Drive all sign BAAs and give you versioning amongst a million other security features. AWS even has offerings that let you take periodic snapshots of on-premise volumes.

Point in time recoveries for the entire account would be nice add too but not having to fork over a million dollars in exchange for a few clicks sounds like a bargain. Hell we were setting up write-only S3 buckets for critical data stores 5+ years ago.

Gross incompetence, I'd fire everyone. A district-wide outage for a week, fine. A 1.4 million dollar check to get data that should have been archived somewhere GTFO.

There are reasons not to use cloud services for backups, not trusting them with your data being a major one. But then you should still be using some kind of internal backup system.

There is no excuse for not having backups.

If you don’t trust them with your data, you can encrypt it with your own keys.

Really, someone just did that for them, for the bargain price of $1.4M. I'm guessing they saved at least that much by outsourcing their entire IT department a few years ago.

This is really the best possible answer. you just did the root cause analysis for them too (you should send them an invoice).

Nothing new, in a way: cut the spending on IT, lower quality, get incidents.

Meh.

But then you lose the main advantages of using them to begin with. It becomes harder to use, it may not be able to do efficient differential backups or snapshots anymore or require you to do some complicated thing to make it work because their interface isn't meant to be used that way etc. And if it's actually important for the data to remain private, you then have to get the cryptography right, not think that your backups are "encrypted" because you used TLS, or use some snake oil encryption software which is using export ciphers or bad random number generation or anything like that.

In-house backups aren't that hard. The hardest thing is to make sure everything is getting backed up that ought to be, and actually test that it is, which is no different with cloud backups.

> But then you lose the main advantages of using them to begin with.

Poppycock. There's plenty of software that can do encrypted incrementals / differentials: Commvault, NetBackup, Veeam, tarsnap, Duplicity, ZFS snapshot send-recv,

You're asserting that these things are as easy to use as keeping all your junk on a shared Google Drive or whatever it is many companies are currently doing?

They all require you to have a level of technical competence equivalent to what would be needed to implement the backups yourself.

If you're talking about something that is worth paying US$ 1M in ransom, then paying someone a few (tens of) thousands in cost centre chargebacks upfront, and probably an ongoing annual fee, is cheap insurance if you want to go with the 'enterprise software'.

And if you want 'consumer software' that offers cloud / offsite encrypted back, then Backblaze offers it in a very clicky-clicky fashion that most folks can handle with a bit of hand-holding:

* https://help.backblaze.com/hc/en-us/articles/217664688-Can-y...

As does Arq, with a documented file format:

* https://www.arqbackup.com/arq_data_format.txt

Every cloud-capable backup system (for consumers and enterprises) does encryption nowadays, so talking about 'data access' in the cloud is a straw man.

If you really want to include NetBackup in that list, then for a real world deployment you'll generally be looking at more than US$1M+.

Assuming it's for an actual enterprise or at least large dept.

Source: Used to be a NetBackup engineer on enterprise accounts :)

I've included more and less cost options in the list to counter the very argument that you seem to be attempting to make.

How much productivity was lost from many dozens of people not being able to work because of lack of access, the IT resources that had to be extended to investigate the various options, and then the US$ 1M ransom eventually paid out.

For Want of a Nail:

* https://en.wikipedia.org/wiki/For_Want_of_a_Nail

eh, there's no free lunch.

“No, we can’t use leading cloud providers to store this data. What if someone unauthorised looks at it?”

This attitude leaves the victim paying millions in ransom. Look, Microsoft and Amazon probably have a better handle on security than your IT dept which is two people who also have to fix any issues like the WiFi not working or software not updating.

There are stories of MSPs who got their offline backup from the same provider who they bought their cloud based RMM tool.

One day an account get's exploited and suddenly the backups are deleted and the systems are all encrypted remotely.

Cloud backups are not offline backups, if you want security use tapes or remote systems which turn on manually.

A backup that can be overwritten is not a backup.

It is possible to use a cloud provider and also write an immutable backup.

> This attitude leaves the victim paying millions in ransom.

Not if they still have backups that aren't cloud backups.

> Look, Microsoft and Amazon probably have a better handle on security than your IT dept which is two people who also have to fix any issues like the WiFi not working or software not updating.

And that's part of the problem, right? Your IT department isn't too bright, so they encrypt their cloud backups with a tool that does the encryption wrong, and the smarter engineers at the cloud provider know that it's vulnerable so now they can read your data when you thought they couldn't. Or you use a weak password for the cloud backup account and then some third party breaks in and gets them.

Not really a concern with a backup tape stored in a fire safe.

There is no reason to believe clouds don't have security issues or software bugs that cause inadvertent data loss.

Just because the cloud provider is a big company doesn't mean the security is better. At the end of the day, they are a team of software engineers with all the usual people and org problems and can have usual human mistakes.

But cloud solutions are better in practice due to totally non-technical reasons. Here's how -

If UCSF can afford to pay millions in ransom, they can definitely afford to hire the right experts to do their IT systems properly.

It is likely that they hire the right people, but then the most commonly recommended security policies were considered but not implemented due to resistence from powerful non-security team members inside the organization. (Usually, the highly paid CISO is playing politics with peers to keep the job by ignoring those security experts).

If it also possible that the IT+security team is incompetent and is only good for deploying expensive but useless vendor solutions that do a lot of security theater but do very little to improve actual security. Also possible that the vendor solutions are susceptible to be misconfigured easily to become insecure.

Public cloud solutions can help overcome these above shortcomings of in-house security+IT teams in a org politics friendly manner.

> Public cloud solutions can help overcome these above shortcomings of in-house security+IT teams in a org politics friendly manner.

The problem there being that the choice of "public cloud" is subject to all the same forces, so then the vendor with the best corporate marketing team gets the business even if their security is crap.

> […] not trusting them with your data being a major one.

Which is why you encrypt the data before sending it over the wire. Just like how you'd use encryption on LTO tapes before sending them offsite.

> I can't think of any reason not to use a cloud hosted service for backup today.

Or just use a non-Windows file server that supports snapshots: NetApp, Isilon, FreeNAS, etc.

If an end-user devices gets infected, and then encrypts mapped drives under the credentials of the user in question, that won't do anything to the read-only data.

Snapshots are not backups, but for a lot of the most common situations, they offer a convenient, quick win so people can move on with their day (often in a self-service fashion by just going into a .snapshot/ directory).

These sound like servers used by researchers. I've worked with higher education research computing and you might be surprised at what you would find.

Researchers may be generating or churning through countless TB of intermediary data, scratch files, etc. Often, the people who actually run the it infrastructure for researchers are... grad students. Sometimes they have grants for hardware and tight budgets, and paying anything for backups isn't part of it. Sometimes, if you're lucky, the it department will be aware of the work and allowed to help.

Now maybe that's not the case here, and there really is one department responsible, and that department decided against spending money on backups. Well maybe they were told by the provost or the dean of whoever that they couldn't afford to back up everything, so they should just stick to file servers. Maybe the boxes compromised here are compute only, and all code and valuable artifacts are expected to be stored safely somewhere else. And maybe the researchers heard this and understood it when they agreed to use the system. But maybe the new grad student didn't get the memo and developed his model in vim on the compute node.

The point is, academic computing is kind of the wild west. Weird fiefdoms and weird restrictions, budgetary and otherwise. It's tough to guess which of these scenarios played out from the outside and we really can't know whether the CISO or CTO, or even anybody working for them, dropped the ball.

I'm so glad I'm not the only one who has experienced this. Only thing I'd add is the lack of understanding about how hard backups are at some of the scales of data in research. Trying to explain to researchers who run the departments that doing a full backup on 35TB+ of data on all budget drives is going to take weeks was something I could never get across, until they lost all their data.

And no, snapshots weren't an option on a legacy clustered filesystem that they wouldn't migrate from.

But that is only like 3x 12TB USB drives.. </sarcasm>

> But maybe the new grad student didn't get the memo and developed his model in vim on the compute node.

Was trying to think through scenario's like that as well, but they don't really make sense.

If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it".

But mgmt decided that work was worth paying US$1.1M+ for, and copping the reputation damage.

So, it's a weird mix of potential scenarios to actually get that happening. ;)

Maybe someone internal was actually responsible for that malware and is going for some kind of weird payday?

> If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it".

Thinking of them as "staff" is the thing. If the random grad student is doing the work the pi really needs done, and deadlines are approaching, then other people's reputations are suddenly on the line. And maybe there's a deadline for a grant that would bring in millions that couldn't be met without that data. Would you pay a million now for much more grant funding?

And maybe that ransom money simply comes out of a different financial pot, but the purse strings are loosened for an important faculty member. Maybe the faculty member is hugely important but runs their own shop for vanity reasons, (which the it department really hates, by the way), but when the faculty member gets in trouble he can pull rank and raid the it department budget for the ransom.

The incentives and power structures in these organizations are complex. And the egos are huge.

Good points. :)

"I'd fire everyone." Too late. They did that 3 years ago. Fired the IT staff and outsourced to India. A great decision that saved them tons of money. /s

https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-worke...

Fire the leadership from the top down. Every one approved outsourcing the IT staff.

From that link:

> This move will save UCSF $30 million over the next 5 years.

So they're ahead?

I'm being slightly snarky here, but also earnest. If they save 30 but lose 1.14 isn't it worth it?

Depends what the total cost was for this...

The 1.14M is just the ransom, doesn't account for how much money was wasted on other aspects of recovery and total downtime...

Substantial reputational damage will have to figure in there somewhere as well. ;)

> ...Fired the IT staff and outsourced to India...

Are you asserting that the breach would not have happened if the IT operations were in-house? Could you also clarify why did you feel it necessary to qualify the location of outsourcing? If the project was outsourced to any other country besides India, this would not have happened?

> If the project was outsourced to any other country besides India, this would not have happened?

I think the emphasis is on the fact it was outsourced to another country; you can't generally check the credentials of someone on the other side of the world very easily. The University of Delhi probably has good IT staff, but “Delhi International Computer Help”? Probably not.

> you can't generally check the credentials of someone on the other side of the world very easily

If that were true, we would not have any trust in global trade. There are plenty of IT companies that have all American IT workforce and still have suffered data breaches. The issue at hand is not checking someone's credentials here. The issue is the political undertone that OP has taken.

Cloud solutions are great unless youre storing huge amounts of data thats being served, modified, and/or backed up. Go with JBOD's, ZFS+snapshots+rsync (or gluster even), and lots of firewalls with MFA and youll be fine (im averaging 60 to 120 TB's in my boxes raid5-7). Google drive is fine for docs. Data should stay home and be backed up elsewhere in the 'house' (still home ... just a separate basement data center or something that wont also burn down). Lets not forget this could have been an inside job that will get around a lot of security other than another humans/montiors vigilance. UCSF outsourcing all their talent is exactly what enabled this.

The paid ransom, will unfortunately embolden the criminals to strike again in search of the next big payday. If it worked once, it could work again.

Optimistic counterpoint: a high-profile, (relatively) high-value ransom payout like USC's may incentivise other orgs vulnerable to this kind of attack to take steps to prevent this kind of issue.

Anything from restricting program capabilities/permissions for external executables, to keeping "colder" backups of business-critical data, to monitoring and responding to software that looks like it's traversing the whole filesystem, could reduce the harm ransomware causes.

You can't really protect against this sort of thing. A lot of our IT security runs on trust. The only way to really prevent this is to make sure that ransom attacks don't pay out.

EDIT: I should mention that I've managed IT services for a major private university earlier in my career, and I am now a software security consultant. When I say it is not possible, I mean that pragmatically. A FAANG company can control their IT well enough to make sure this doesn't happen to them, but a hospital or university relies on computer systems running software way outside of their control. That MRI machine? Its controller is probably running some ancient version of Windows Server 2003 with proprietary drivers. That university registration app? Custom coded by generations of CS student interns running on a shared system whose operating constraints are set by the Novell GroupWise instance that is co-hosted on it.

As a practical matter, one of these organizations simply cannot reduce their risk to zero or near zero. There's too many attack vectors they don't have control over. The IT departments can't mandate proper security because they don't have the budget to enforce.

You can't fully protect, sure, but you can have a person or team alerted on suspicious behaviour. You could also try to configure your infrastructure so that any code imported via vectors that ransomware usually uses is compartmentalized in a VM, container, or other chroot-like env.

And honestly, having even week-old cold backups makes this kind of attack _considerably_ less scary and cheaper, and it enables you to skip the payout (and I'm on the same page as you on that — if there's no money to be made, ransomware attacks will drop off).

  like USC's

UCSF is University of California, San Francisco. USC is University of Southern California, a private school.

Good catch, thanks!

If the data is worth paying a million dollar ransom to unlock, it is worth setting up proper backups. I for one am grateful to people who commit these crimes in which they "lock" data in place rather than sell it to the highest bidder.

Proper data hygiene isn't brain surgery. There is zero excuse for this event. I don't blame the criminals. I blame the university system. Shame!

Depends, those kinds of outfits tend to poke around and lurk a while before striking. In that time they can exfiltrate and you cannot prove that the baddies didn't exfiltrate data (if you had that sophistication, they wouldn't have been in the mess they got into).

I know how much IT personnel at UCSF make -- you get what you pay for. If you want expertise, it's not hard to find.

People who can properly secure a large enterprise are, actually, quite hard to find.

They're really, really not. You just have to pay an appropriate salary.

...and allow IT to control the security and access policies, rather than executive level users.

Or the faculty senate, as is often the case in higher ed.

Don’t entirely disagree, but also think it’s fair to say they almost certainly use the stolen data to find weaknesses in their next targets, so it’s not just a one to one thing. This doesn’t at all negate the main statement: good motivation to actually do proper backup and security.

    ... "lock" data in place rather than sell it to the highest bidder.

Why not both? And once the rightful owner of the data has paid a fat ransom, surely that's got to provide some kind of proof of its market value. The University did say that

    The attackers obtained some data as proof of their action

so unless they're logging their outbound traffic, who's to say they didn't exfiltrate all of it? It's the kind of thing that the University would remain tight-lipped about unless they were either sure that it hadn't happened (doubtful, seeing as they aren't running a tight ship) or had some kind of mandatory reporting obligation for the data.

The data is worth that much to the university because they're critical to grant continuity - it'll be hard or impossible for their researchers to keep the money flowing without it. It's pretty much useless in everyone else's hands because those grants also depend on individual reputation and research history.

Absolute nonsense.

First of all, they are increasingly selling the data. They exfil first, lock second.

Second of all, these wonderful criminals are targeting all manners of institutions, not just large universities.

Proper data hygiene at large enterprise levels is, in fact, exceedingly difficult.

  Proper data hygiene at large enterprise levels is, in fact, exceedingly difficult.

Creating a hermetically sealed IT environment where only way to exfiltrate data that remains is the employees eyeballs is definitely possible and is increasingly done well by a lot of large organizations.

Defending against insider threat (malicious employees) is still a challenge for most civilian (non-military) organizations.

This was a bug bounty - another way to look at it.

Maybe there should be a law that if you pay a ransom, you are required to pay the same amount as a fine. Because paying these ransoms is funding the criminals.... how about you have to also fund law enforcement to combat those criminals?

(also, this should reduce the amount that actually goes to the bad guys, since the amount of ransom would have greater downward pressure, i.e. if they'd probably not be able to collect more than $0.57M because that would cost UCSF $1.14M)

This would make the payers far less likely to report it, and ultimately make it much harder to track

Well if it was illegal to not report it, they'd be putting themselves at a lot of risk. I can't see a university or corporation making a large payment like that illegally.

Also, it may not make the costs to them more. Remember the amount of ransom is based on what the ransomers think the ransomees are willing to pay. Today, the reason it was 1.4 million instead of 2.8 million, is that they didn't think UCSF would pay the latter amount. So if they knew UCSF would have to pay double the amount of the ransom, they'd have to only ask for half as much.

Or since we already have to pay for law enforcement anyway, they could just do their jobs and catch the bad guys. But I think we all know at this point that "catching bad guys" is just a pretense, right?

Not sure what you are saying. We already pay for law enforcement, but their resources aren't infinite.

This would serve three purposes: help fund those things that are costly, deter the bad guys (since they can't ask for as much money if paying a ransom is going to be twice as costly to their victims), as well as to add additional incentivize people to secure their systems.

I don't know what you mean by ""catching bad guys" is just a pretense". Pretense for what? By who? That sounds very conspiracy minded.

> you are required to pay the same amount as a fine

Pay a fine to whom? UCSF is a state institution.

Many of the other recent examples are cities, counties, and so on.

I've worked in an institution for which I rewrote a public facing database to replace their old and clunky system. I had numerous emails with them to transition smoothly between the old and the new system, and the last step was for them to give me the latest snapshot of the MSSQL db that would allow me to import the last two weeks of data entry. Scripts were ready, so the down time would be 10 min which was totally acceptable for that tool. That's when I realize the backup was two weeks old, that they had already deleted the machine as soon as the transition started (against our plan), and that the automated backups at the scale of the university had not been working for months (they realized it because of this incident)...

What kind of data is worth at least a million dollars and isn't properly backed up? Unbelievable. Some heads should roll .

It's a university, university IT is often ultra political and those who win the various battles make the rules, regardless of competence... and often without IT's sign off.

Some universities generally have done better about such things and are making progress... but generally there is a push and pull for IT dollars by unversity departments who want to spend that money as they wish for their given programs and then that money comes FROM IT ... who down the road are then tasked with the costs related to maintaining it and the terrible decisions a department made in the meantime... or in the worst of cases tasked with securing that data and / or making it work at all.

It's the same story for IT in the private sector to some extent, but it is way worse at many universities. Imagine if your HR director got to pick the PCs to support, networking equipment, software, backup methods (if any) all on their own and wanted zero input. That's kinda how it is at many universities.

I spent months helping a large university dig out from a program where they hooked up some super special microscopes worth millions of dollars ... to low grade network switches and storage. I got to try to explain why you can't put 10,000 pounds of data into a borderline consumer grade network ... in all of a couple milliseconds.

> It's a university, university IT is often ultra political […]

While you're not necessarily wrong, another option is budget.

If this is academic- / research-generated data, then it could have been paid for by grant money, and most of the cash goes to paying grad students and perhaps some computer equipment.

IT may have chargebacks (they have bills / cost centres to pay too after all), and no one wants to "waste" grant money. Often these things are 'shadow IT' run in an ad hoc fashion by just throwing together some PCs.

If the group's expertise is in medicine / biology, how many members want to give up their day-light research hours to run the computer infrastructure?

I've spent about half my IT career in the academic sphere, and cheap solutions can be a fight to implement even if they solve the problem; even free (open source) ones can be an effort if they take time or slow down the workflow.

And these people aren't stupid: they 'know' they should do some of these things. But people 'know' they should get exercise, and how many folks do that?

> academic work

I actually know a few people working on their PhD’s that have nothing backed up.

Their advisors buys servers, the university’s IT sets them up, they get access to it and work on it without backups, local copies or anything.

They have grants that they may have to provide results for.

As time passes, there are more people working in that lab that get access to the machines. Multiple projects being worked on for multiple grants.

I also know people running unsecured stuff on public ports for easier access.

Another thing, it could be that the data sets they where working on where given to them under a condition that they wouldn’t be shared, exposed or something.

I don’t know anyone in medicine or healthcare and no one at this university, but that’s what I’ve seen from people I’ve met throughout the year.

(I have also seems people loose their thesis because their laptops died on them and they didn’t have a backup or for their Word file getting corrupted.)

What I find crazy about this -- no guarantee that the ransom payment would unlock the machines -- did they send 1.14M in one go or was it a smaller amount for the first machine, then an additional fee for each additional machine?

Also would be interested to know -- was it Bitcoin or some other cryptocurrency that was used?

It’s in the best interest of the people who make these types of malware to provide the decryption tools once the ransom is paid.

If news gets out that even if you pay the ransom you won’t get the decryption tools then no one would pay the ransom and the hackers get no money at all.

Apparently crypto-ransom people are actually pretty trustworthy about unlocking the machines. It doesn't really cost them anything (0% chance you were gonna send a 2nd payment if they didn't unlock), and their reputation as 'fair' is very important for securing future ransoms.

Someone's gotta be thinking about doing a ransomware operation that doesn't unlock the data in order to poison the well.

It's been done. There was one "ransomware" attack that just erased everything.

NotPetra

That creates an interesting grey hat scenario where a normally good actor could execute a few ransoms and not unlock the machines. Eroding trust in the entire scheme could reduce the long-term effectiveness of the scheme

I have heard that they have really good customer support once the payment has been made.

As weird as it sounds its pretty much in their best interest to send the unlock key because if they don't then people are going to stop paying them.

The FBI's last attempts at breaking crypto rings found that these guys had 24/7 technical support and a phone number you could call that would actually have a person on the other side.

Friend of a friend said it was probably the most "customer" focused organization they've ever fought against

> The data that was encrypted is important to some of the academic work we pursue as a university serving the public good. We therefore made the difficult decision to pay some portion of the ransom, approximately $1.14 million

Once you’ve paid the Danegeld, you’ll never get rid of the Dane.

I recently listened to Mikko Hyppönen as the company I work for had invited him to give a talk. It was very interesting.

One thing I hadn't realized before was that ransomware criminals has developed their own backup strategy:

- in addition to encrypting the data they will also exfiltrate it and threaten to publish it ob the internet for everyone to see.

That way it doesn't necessarily help an organization just because they have multiple layers of offsite read-only backups.

How did they not have backups of important data?

there was mentioned a threat of publishing of some secret data, and that i think may be a reason why UCSF was willing to pay

https://www.trialsitenews.com/hacking-group-launches-success...

This seems much more likely than the scenario where they simply didn’t have any backups of the data.

Did you ever work for an organization?

It's amazing how bad some of the practices are outside of tech companies. I worked for a pharmaceutical company, one of their half million dollar microscopes was saving data locally, when I asked if it was backed up at all the answer was: "no ... but it's using one of those red hard-drives".

What a cheesy attack. It only cost them $1.14M to get their data back. I can not tell if the attack hit something inconsequential, the criminals are stupid, or they just do not understand finances for them to ask for such a tiny sum.

UCSF received $1.43B in grants and contracts during 2017-2018 [1]. Assuming they are generating an equivalent amount of value in knowledge evenly distributed over time, the loss of one day of research would be ~$3.9M. So, if the the last whole organization backup was one day ago and the attackers were only able to stop access to the last day of work since they did not think to corrupt the backups before they went out, then the ROI of paying off the ransom would be ~3.43. If they were able to affect the entire organization for an entire week, then the cost would be ~$27.3M with an ransom ROI of ~24.

So, assuming they did any damage of consequence, asking for ~$1.14M seems like robbing a person at gunpoint for their pocket lint.

[1] https://www.ucsf.edu/news/2019/02/413396/ucsf-top-public-rec...

Doesn't matter, it's the perfect crime...

The encryption is done on the user's machine using their processing power, and there's virtually no downside for you (either they pay and you decrypt or they don't and you just dissapear).

Ransomware will be an issue for years to come.

I am confused by your response since it seems to agree with my post but you used words that indicate you think that they do not agree. Did you mean to reply to someone else?

I am saying that that the ransom is hilariously small compared to what they would probably be able to get. So, when they realize they can actually ask for a number that is not a rounding error they will probably increase the number/outcome of attacks. They are criminals robbing people at gunpoint and only asking for their pocket lint. Once they all realize they can ask for a wallet and get it, I think the number is going to go way up.

On the other hand, the more they ask for, the less likely they’re going to get paid for it.

If they keep asking for $1M from organizations that will pay because that’s pocket change, the more organizations they can attack.

If they start asking for too much, they’ll see they won’t pay as quickly, other people will start selling services to protect against you that seem more attractive, etc.

You make the assumption here that this hack halted all research. It's unlikely to have even affected 10% of ongoing research. Plenty of things stored on cloud storage, in email, in personal computers, on computers not locked out, etc.

The article is inconsistent about the severity, which lowers its trustworthiness. It states that everything is under control, a top super leading security guru expert is on top of it, no important data was stolen or exposed, everything will be fine soon. And... oh yeah, BTW, we'll be paying an astronomic sum of money to the criminals to get our data back.

Wonder if the cost of backups for the affected servers would have been less than the US$1.1M they paid? ;)

At a guess, they could probably have put in a some fairly high quality storage + backups for that price, and still had money left over for all expenses paid staff vacations. :)

Is there a way to restrict encryption at a hardware level? The conditions where you would like to voluntarily encrypt data are usually quite rarefied. Allowing any sort of encryption activity on your system seems like a hazard these days.

Storage systems will just see a lot of data being changed.

You could have some sort of alert in place, but if the malware doesn't write fast enough to trigger it, you'd still miss the problem.

Generally, the approach places take is having backups (eg to tape, off site, etc), and/or having storage that makes a snapshot every few hours and retains them for days/weeks/months.

Snapshotx are generally very low cost and easy these days, as it's just a pointer manipulation thing on (say) ZFS rather than a complete copy of the entire data set.

Backups are the default solution. But my understanding of this UCSF case is that their system was compromised and then ransomwared. In this situation, the attacker could interfere with your backup system for several weeks or even spoil existing backups.

It would still be nice to restrict encryption on a system that you control. I suspect, but don't know, that encryption has a particular pattern of memory and CPU activity that could be recognised. Or if there are a few commonly used libraries you could do it that way, although an attacker could roll their own encryption.

To understand why they didn't back up: a) they are academics, which means funding is lacking and b) it's in the field of medicine, known for its lack of tech adoption.

Here is Dutch University of Maastricht paying 0.2 million euro ransom last year in December.

Seems like business is blooming.

https://www.reuters.com/article/us-cybercrime-netherlands-un...

Don’t they have insurance for these things? A small college near me had an attack like this but paid via insurance.

Would you want the insurance policy to pay out though?

At some level of recklessness, insurance becomes void. I think a lack of infrastructure to restore a hacked server — with data valued at over $1M — is negligent enough to not be covered.

But maybe UCSF are on MegaCo’s YOLO tier of server insurance, which is so expensive and isolated it has no impact on my MegaCo pet insurance premiums?

But patients suffering for administrative negligence? I don’t agree with that but there does need to be some kind of incentive there.

That’s a good point, I hadn’t thought of that.

Perhaps the health of some businesses is so important that they should be protected by, ahem, state-backed insurance paid for by the taxpayer?

I'll venture a guess that UCSF was nominally in compliance with some relevant bureaucratic regime (ISO 27001, SOC 2, etc), and that was good enough for a stodgy insurance company that's not very sophisticated about "cyber risk" (in case use of the term "cyber" isn't a giveaway...)

The CISO’s top 5 “security tips” PR piece didn’t mention off-system backups. Policy voided.

Most of the time, the insurance guys will just negotiate on your behalf (lowering costs) and handling purchasing of the crypto.

They also won't tell you how much they paid out...

I'd expect that UCSF is self-insuring.

How would that work? Would the bad guys just ask for 10 times as much, since the insurance company is obligated to pay whatever it costs?

If kidnap & ransom insurance is any precedent: the policy has a limit, the insurance company's business is negotiation, and the whole thing is shrouded in secrecy to keep attackers from knowing who is insured & for how much. Even attackers who get paid have an incentive not to piss off the insurer.

My school district has insurance for this, and we paid a ransom (via insurance) a few years ago.

This just illustrates how incredibly important solid backup strategies can be. A big university should be able to figure out how to make WORM (write once read many) backups of their data. A million bucks buys a shitload of cloud storage or physical airgapped tapes...

The same weakness that resulted in lack of backups may be related to how the intruders were able to get onto the machines in the first place.

How incredibly negligent and incompetent of the ucsf IT staff to not have this data sufficiently backed up, not to mention protected. Does this university offer IT courses? If so their reputation has been seriously tarnished.

I have to assume things were backed up, and the attackers were deep enough in the system to find or delete the backups.

Anyway, I wonder how this payment was made...

I'm surprised that paying the ransom is a thing.

This is a very selfish thing to do, as you encourage the authors of such attacks.

Ouch. No backup?

It's pretty obvious that the answer is 'no backup'...

It’s not that obvious. It could be assurance that the data would be deleted and not sold/shared.

> We therefore made the difficult decision to pay some portion of the ransom, approximately $1.14 million, to the individuals behind the malware attack in exchange for a tool to unlock the encrypted data and the return of the data they obtained.

I assume you read that though before you replied, right?

> and the return of the data they obtained

… as if you can’t “return” the data and keep a copy at the same time …

Might require ‘best effort’ to keep data private.

As a reminder, in 2017 UCSF offshored all of its IT staff to HCL Technologies and forced their then-employees to train their replacements before laying them off.

They brought the replacements into the Bay Area on H1B temporarily while they were trained by their soon-to-be-laid-off counterparts and then sent back overseas to continue their roles once training was complete.

https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-worke...

What kind of monster of an employer makes their employees train the replacements they're getting fired for?

If that were me, I'd organize and have everyone quit; let them figure things out. Screw the pittance of a severance.

Their severance pay was probably on the line...too sad

Disney did this same thing: https://www.orlandosentinel.com/business/tourism/os-bz-disne...

From what i have seen over the years this is an industry standard practice. Don't assume the employees about to be sacked are in the know.

Sometimes this is hidden under the guise of big IT modernization projects. The company transitions or consolidates towards ERP platform X. It is a huge operation that touches almost every part of the business. Armies of analysts are brought in to map the AS-IS business operations, the system is developed, once stable enough, the other shoe drops and the whole IT department is now moved off- or near shore to the company that transitioned the system while other 'non-strategic' parts of the business are 'streamlined' into managed service contracts with other parties in the project consortium.

> I'd organize and have everyone quit; let them figure things out.

Be careful here... due to "freedom to choose" legislation targeting unions, it may be illegal for you to "organize labor" in your state...

Heard firsthand from employees in the pharma industry that were forced to train their replacements that would do the work in India (quality control of medical devices). If they didn't comply they would get fired immediately and not get a special severance package... most of them did comply and trained their replacements because the job market was bad.

Someone, somewhere, is calculating whether the ransomware line item was worth the offshoring and laying off local employees line item.

What would it mean if it was?

While what they did was may not be right, is there any direct link about this to work done by HCL? Have all of such previous attacks been linked to work done by H1Bs?

How the heck did they qualify for H1Bs when they were replacing already employed Americans? Not only did the H1Bs unemploy citizens, we didn't even get the long term benefits of trained workers living and paying taxes in the States.

using H1B for 4 months stint is very strange. It is too valuable a visa slot (significantly oversubscribed lottery usually, at least in the recent years) to be wasted that way. Usually it is done using something like B1 for such a short trips, especially if it is training, etc. I wonder whether the journalists and others did mistake one visa for another, especially given that they naturally wouldn't be privy to such internal details of a 3rd party company like the HCL in this case.

I looked a little more and found this subsequent article talking about it:

https://www.latimes.com/business/hiltzik/la-fi-hiltzik-uc-vi...

My understanding is that H1-B's for some educational institutions (and certain affiliated non-profits) are cap exempt.

but they did get the benefit of cheaper IT workforce? What they did to their existing employees was certainly morally wrong but is outsourcing your IT support to cheaper third party seen as morally wrong too? Isn't that the idea of free captial market? Sure the state didn't get taxes from those employees but state owned university saved costs.

They did benefit from the cheaper workforce, in fact they saved 1.14 million dollars!

Wait

Oh no...

You say that as though 1.14 million dollars is a lot.

swing and a miss

The H-1B program is not intended to give employers access to cheaper foreign labor. Its stated purpose is to allow people to immigrate when they can do jobs that US workers can't. When there are already US workers doing those jobs, that's clearly not the case. But that sort of abuse has been all too common.

The recent H-1B suspension probably goes too far the other way -- the program needs more safeguards against abuse rather than to be discontinued -- which requires a legislative solution (auctioning off the slots seems promising). But the legislators are apparently too busy flinging their own scat at each other right now because it's an election year. In the meantime it might not be such a bad thing to fail closed rather than fail open, given the effect of the lockdowns on the unemployment rate.

Because HCL employed them, not UCSF.

A large percentage of H1Bs go to "bodyshops", which import workers from abroad to temporarily work at companies that are offshoring their employees. Yes, it's perverse; they are in effect being used to replace American workers, but not explicitly.

The H1B system is totally out of wack. One of the many things that we need to tear down and rebuild from scratch in 2020.

Lots of people are brought from overseas on B1, J1, L1 whataver cheap visas and employed illegally. All these HCL-Wipro-Mahindra rent out whole apartment complexes in the areas where they have large customers; there these B1s are allowed/forced to live on cheap.

It's tiring to keep hearing these stories and tieing them to H1B. Satya Nadella also started on an H1B, Sundar Pichai did too and so did Andrew Ng. Just those 3 combined have created more jobs than were lost here. So please, stop spreading partial info that creates hate against a whole swathe of people who have come here legally, have contributed extremely productively to this nation in the form of taxes and labor, and would like to be treated with atleast the same level of dignity as any European low skilled person who just got off a boat at Ellis Island.

I think most of the sentiments here are regarding the abuses of the H1-B program by the American companies and not targeted towards the recipients of the visas themselves. I don't think it's reasonable for people to draw the conclusion that the normal people that were awarded the visas should be to blame.

I do believe that you perhaps aren't tieing the program to the recipients. But I don't think the distinction between the program and the beneficiaries is clear to the average voter who likely hasn't ever met a person on an H1B. When they read these stories their instinctive reaction is jobs going to "other people" and then they vote for people who are against immigration, who are tacitly for xenophobia. You could have called me pessimistic, but we have clear examples now of how xenophobia can resonate and can be built on labor issues.

I find it difficult to believe that we read the same article.

Based on the specifics of my comment and the context of the article at hand, I don't understand how your point ties to my comment and do not appreciate the insinuation it brings.

This is a labour issue, not an issue of xenophobia. Though I'm sure these companies would love us to believe it's about xenophobia. The breakdown of western labour protections and the march towards near chinese levels of exploitation will likely be overseen by people who claim that western labour protections are racist.

"They are taking our jobs" is also overtly about labor but the jump to xenophobia happens pretty quickly.

It wasn’t all IT, not even half.

H1B for outsourcing companies has created massive arbitrage opportunities for American companies (primary non-tech). They find it cheaper to off shore their IT operations. H1B has a lot of "loop holes" that benefit employers. They're as follows:

1. H1B ties employees to a specific employer. This bonded labor prevents free labor movement. This allows the employer to exert tremendous control over the employee. Free labor movement is important to discourage employers from depressing wages for American workers.

2. H1B is allowed to participate in the EVC model. This would be fine if the H1B was not tied to the employer and instead it belonged to the employee. In absence of this, it explicitly allows exploitation of H1B workers hurting American employees.

3. H1B is dual-intent non-immigrant visa and is the first step to permanent residence. The employer dangles this carrot and exerts further control over employees. The long wait times (on the order of a decade or longer) due to the greencard backlog for certain countries (primarily India, China, etc.) make it worse. More importantly, the employer can rescind the application throughout this process and completely destroy the employee's professional and personal life.

4. Outsourcing companies will not start the greencard process for H1B employees until they're in the 5th year of H1B. Combine this with the 10+ years wait for certain employees, ties the employee to the employer for at least 15 years.

5. Until recently, employers could've rescinded the H1B any time and the employee had to exit the country immediately (no grace period). It was only recently that the US government gave a 60 day grace period for H1B visa holders. Imagine how a human being would feel if they have lived in the country for a decade, making their lives here, having to wind up and leave within a few days or even 60 days. Its simply inhuman. However, this is where we are right now.

This issue is more nuanced than it looks like but the underlying problem is due to the fact that the H1B visas and permanent residence process is tied to employers. If we look at visa programs in pretty much any other country, visas belong to employees, their PR process is between the government and the individual applying for the PR. The employer has very little control over the employees. This is why such large scale exploitation doesn't exist in other parts of the world.

There have been attempts over the years to fix this issue. The infamous Neufeld memo tried to eliminate the EVC model. Unfortunately, that did not fly and was eventually rescinded. Honestly, a tiny percentage of H1Bs are used by tech companies that genuinely treat their employees well. If you eliminate the bottom 80% of the H1B employees, EVC companies will shut shop and you'll also hurt foreign new grads.

Reply on news.ycombinator.com