Hacker News Reader

Comment by drhagen | Hacker News Reader

I worked at Merck for three years as a scientist and only left a week before this went down. My former colleagues said they stood around and did absolutely nothing for days and then struggled to get the tiniest amount of work done for weeks.

The article chooses not to get into stunning mistakes by Merck's IT that allowed this to happen in the first place. The patches for the EternalBlue exploit were released by Microsoft on March 14, but Merck's IT chose to sit on it for over three months. (Like many large companies, they disable Windows update, choosing to release patches on their own schedule.) Even after the WannaCry attack crippled computers around the world on May 12, they still had a month before NotPetya brought them to their knees on June 27.

View on HN

While patches would have helped in this specific case, that's only because Merck was collateral damage.

In a targeted attack, it's likely the foreign agency would be using a 0-day attack.

The only way to protect against that is by reducing the OS monoculture, offline backups, and using network air gaps on critical data.

But those practices are extremely rare in my experience.

If I was on unfriendly terms with the US, I'd use this as a case study on how to cripple the economy by taking advantage of the large monocultures created by lax IT in a hundred or so of the largest firms.

> In a targeted attack, it's likely the foreign agency would be using a 0-day attack.

A targeted attack is also expensive and the victim would need to have something worth this kind of money and attention. "Nation state actor" just isn't a reasonable risk assumption for a great many organizations.

> The only way to protect against that is by reducing the OS monoculture, offline backups, and using network air gaps on critical data.

When the "nation state actor" comes looking for you with some motivation, all that and the air gap won't mean much. See Stuxnet.

Like J. Mickens said: "Basically, you’re either dealing with Mossad or not-Mossad. If your adversary is not-Mossad, then you’ll probably be fine if you pick a good pass-word and don’t respond to emails from ChEaPestPAiNPi11s@virus-basket.biz.ru. If your adversary is the Mossad, YOU’RE GONNA DIE AND THERE’S NOTHING THAT YOU CAN DO ABOUT IT."

https://www.usenix.org/system/files/1401_08-12_mickens.pdf

Nation-state actors can be deterred by nation states. If Vova believes that CNAing someone in the US will cause the US to bankrupt him and/or the people whose support he requires to stay in power, he'll make damn sure this doesn't happen. As long as the US does not demonstrate this capability and willingness to use it, he'll continue to misbehave.

Offline backups are about the easiest thing you can do, and they protect against pretty much everything. Air gaps are useful, but they're just a connection with unusually high latency. Network monitoring protects against zero-days.

The fact that good is worse than perfect does not mean good is no better than bad.

Having every machine in the company three months out of date on critical security patches is just negligence. I'm surprised the insurance companies didn't take that tack.

0days are precious and expensive. The odds of being targeted by one are incredibly low compared to those of being targeted by an exploit for which there is a patch.

True as that may be, let's keep in mind just how juicy of a target Merck is, being a gigantic multinational with a market cap roughly equivalent to the GDP of Portugal.

Is this related?

Merck has a new IT Head - joined on Nov 2018. The attack happened on Jun 2017 (i.e., 1.5 years earlier). Jim Scholefield - https://www.linkedin.com/in/jimscholefield/ Great pedigree: Nike, Coca Cola etc.

[Edit]

Seems to be: He will also have oversight of cyber-security – a big issue for the company after a ransomware attack in June 2017 brought the company to a grinding halt. Scholefield will be part of the company’s executive committee, reflecting how integral the digital transformation drive is to the business.

http://www.pmlive.com/pharma_news/merck_and_co_picks_nike_ex...

Probably. The whole time I was there, IT was a disaster. I did not know if the people at top were incompetent or they were just woefully underfunded like IT is in many companies. After a billion dollar loss, I would hope Merck came at it from both angles just to be sure.

My favorite memory was a mandatory security training for all employees. They had a couple of slides on how to make a good password, and one recommendation was to use "keyboard encryption". This is a technique to take a bad password like "ClevelandIndians" and shift the keys to the right (or other direction) to get "V;rbr;smfOmfosmd", a supposedly better password. I stood up at the Q&A time and "asked" how this meaningfully improved passwords given that it added at most two bits of entropy. I also responded to the "how was the training" survey with a recommendation to teach people correcthorsebatterystaple-style passwords instead. Colleagues who had been assigned to a later session said that a slide containing the XKCD comic had been inserted into the deck.

"correcthorsebatterystaple-style"

Are you saying those are better than the 'keyboard encryption'? Because they're not, every password cracker has functionality to string dictionary words together in various permutations.

Yes, they are (assuming the words are actually chosen at random).

The idea of "correcthorsebatterystaple-style" passwords is to randomly choose 4 words from a pool of about 2000 words. That gives about 11 bits of entropy per word, for a total of 44 bits.

With a 2-word "keyboard encryption", even if you choose the two words the same way, you only get 24 bits of entropy: 22 bits for the words plus 2 more bits for the choice of which direction to shift (up/down/left/right = 4 options = 2 bits).

yes, it's mathematically better to have longer passwords than more complex character sets.

Dictionary has a lot of words. Even if you knew I chose 4 of them, gonna take you a little bit of time to get through those combos.

I'm not familiar with their environment but depending on the software and vendors who support aspects of software, those patches may be held at their request. That is people like Rockwell, Emmerson, whatever, may not “release” a patch because it can have implications for GxP environments. So not saying this is the case, but there are times when that is the case and companies have to sit on fixes.

However in these situations those systems are siloed and segregated do that things don’t propagate. I have no idea how Merck is setup.

Unless you do development where a Windows patch could break a complex environment, most people in the workplace are always using all Microsoft products anyway so they should just be on auto-update. All it takes is for some IT manager to sit on a critical patch for too long. If auto-updates break your setup, then you could opt-out and be moved to a sandboxed environment where you still get patches but only after they are verified. I remember when some vcredist patch broke a very expensive development suite. Despite all the engineers affected complaining to IT, it took them weeks to roll it back for us. In the mean time, we had figured out ways to debug the tool with Visual Studio, catch the error, and continue past it without crashing everything. The patch must have broke quite a lot of things because there was another one that came shortly after that seemed to avoid the problems.

If you’re in a large organization you do not want an auto-update to mess up something for 10,000+ people who are unable to work because the VPN or whatever other enterprise system/software stopped working.

I’ve seen enough issues this year with Azure and multi-factor sso being down. It makes me weary of Microsoft’s updates. Lots of customers screaming because they can’t access our portals.

Sometimes your vendors have to wait for Microsoft to fix something they broke which complicates it even more.

In many cases unpatched systems automatically fail GxP by not being patched but pharmaceutical organisations still run operations like it's the 90s and they just don't acknowledge the problems. Have worked in pharma IT for 10 years.

I think the problem is that their vendors’ applications run like it's the ‘90s. Often the orgs will be waiting on a vendor’s patch to be released which has been qualified for the sec patch. This requirement is kind of dubious but if you patch before they release their patch you’re on your own.

You'd think so but most vendor security patches appear very quickly even for 90s style systems, the problem is almost always the customer's own processes. All vendors in the pharma business space maintain a dedicated support and patch team for all deployed and commercially supported products and or course charge customers for the privilege.

As far I understood, particular machines were under GxP requirements, but the vast majority were not. We scientists had local admin access to our laptops. To access some GxP software, we connected via a client that was like a remote desktop.

> One researcher told a colleague she’d lost 15 years of work.

Either IT or this person is grossly incompetent. Beyond patch policies, managing data this way is terrifying.

Reply on news.ycombinator.com