Hacker News Reader

Comment by threatofrain | Hacker News Reader

The hospitals, airlines, police, etc. that bought into CrowdStrike should have primary responsibility. I wouldn't want to live in a world where they can say "but McKinsey said it was okay" and wash their hands of this, while pinning all the blame on CS.

View on HN

I don’t get it; in most professions, if you deliver crap, you are liable. Why is this different? How is the onus on the buyer who got told this product protects them and then the product actually screws them? I think CS should pay damages and go bankrupt, so others might think twice in the future.

I wouldn’t want to live in a world where people can get away with mistakes like this: as far as I have seen, if one person in CS had actually tried this file on their local windows machine, it would’ve crashed. That’s not a small thing is it? That’s a Boeing like culture which should be terminated today.

It is one thing if these updates were in hands of the IT of the companies, but that this company just could automatically push them over is just weird.

I accept shared responsibility when there is some agreed deployment and such scenarios, but here? Nope it is on CrowdStrike...

If you hand your car keys to a drunk driver, it’s on both of you.

If I give root to a third party, then I’m liable for everything they do with that root.

The guy was not drunk when the key was given.

I’ve seen crowdstrike’s process. Seems drunk to me.

If I hand over my car keys to the same cleaning companies every month and all is fine always, except this last time, the cleaner cuts the break lines while cleaning the under side and I die in a crash.

That is a nonsensical comparison. Remote software updates and administration are common professional services that consumers of software cannot execute themselves.

The correct comparison would be handing your house keys to a licensed insect exterminator and the guy accidentally poisoning your family. You of course are not liable if a professional supplying you with a service messes up.

Crowdstrike doesn’t test their releases.

It’s like paying a licensed insect exterminator that uses random chemicals. They tell you this and you hand over the keys anyways.

An other view. Tesla does over-the-air updated you don't verify. It disables break, you drive car and kill someone due to this. Do you share responsibility? What if this happens while driving?

It was a virus definition update, not a software update. CrowdStrike will obviously update their virus definition deployment pre-flight check to prevent a situation like this from happening again, but a staged roll-out is out of the question. What do you want them to do? Expose everyone to new known viruses for longer?

> CrowdStrike will obviously update their virus definition deployment pre-flight check to prevent a situation like this from happening again

I don't consider them capable of preventing something like this from happening again.

No worries, it will happen again. That’s why they need to be euthanised.

> a staged roll-out is out of the question. What do you want them to do? Expose everyone to new known viruses for longer?

I think trading, say, small x (15 minutes, an hour?) time, to stage rather than blast might be an acceptable tradeoff given the visible consequences of not doing so.

OTOH installing insecure/unstable “security additions” on top of windows should be consider a terrible idea (ie making a bad situation worse).

Our essential services should be built on a more secure OS, if windows isn’t suitable.

> OTOH installing insecure/unstable “security additions” on top of windows should be consider a terrible idea

So it sounds like you agree that Crowdstrike should go bankrupt.

> Our essential services should be built on a more secure OS, if windows isn’t suitable

Oh wait you blame Windows for this?

When an IBM mainframe was the routine system, well recognized as a single-point-of-failure, it wasn't a problem like this.

Hate to break it to you:

- https://www.crowdstrike.com/blog/crowdstrike-brings-xdr-to-z...

- https://access.redhat.com/solutions/7068083

smh, big iron is in the crosshairs and Linux to boot.

Or maybe not to boot . . .

> Why is this different?

Because critical infrastructure doesn't get the convenience of shifting the blame. They must account for these risks in advance. Do you know what happens when IT breaks down in a bank? They've got two or more printers that print all new transactions. Database corruption? No problem, we've got the transactions right there. Printer breaks down? No problem, we've got another one printing the transactions right there. Hospitals don't get the luxury of shifting the cause of death from them failing to properly account for risks to a piece of software.

I think they need to do both. But then again, I don't think most of the IT staff in these places are very capable. I have large hospitals as clients and everything is fairly terrible there (it is how I make most of my money; I solve these type of issues for companies when shit hits the fan and hospitals are kind of the worst, but banks or gov orgs are also not great). They don't understand any of this really so it's not weird that they buy stuff that's recommended in CIO magazine. But yes, they cannot shifting cause of death indeed.

Where is the line drawn on how much is the responsibility of the "end users" (hospitals, etc).

Hypothetically, if there was a widespread 'bug' in... Intel CPUs that caused a similar issue, would you say it was the hospitals that should be held accountable, or Intel, or someone else? What about if the issue was in Windows itself - what should each hospital do differently that would have avoided this?

Let the consumers sue the hospitals and banks, and let those entities that directly interface with consumers sue Intel in return. Consumers need an interface to recover damages. This will provide the motivation not to keep around shit vendors and entertain shit contracts that basically absolve the vendors of all blame.

When the damages are in the billions we need to make sure that there are billions in liability to balance the scales. Or else there will always be a perverse incentives to not give a shit.

I think BOTH must be held liable. I’d share the blame 50/50. And if Microsoft should have checked the CS sw because msft signed it (I’m not clear on if that happened), then I’d generously make msft also 10-15% responsible.

Hospitals bought into a contract where the terms basically said "don't rely on us for anything important, especially not anything that impacts human life". Patients were certainly medically impacted, perhaps up to the point of early death.

Whose incompetence is that? If a hospital pays for an energy redundancy solution where the vendor basically communicates "please don't use us for anything important", and then a power outage costs lives, are we seriously saying that the hospital shouldn't have to pay big time?

I'm not saying that CS shouldn't be liable. I'm saying that when consumers are looking for someone to sue, and regulators are looking for someone to punish, those who directly interface with the consumer should be in the direct line of fire. Hospitals in turn should be able to sue CS.

We certainly should not prefer that the blame goes straight to the engineer as then the consumer would basically have nobody to sue. That's a world of even less accountability. Those who are most capable of taking responsibility should take responsibility, and not merely those who are most blameworthy from first principles of local causality.

There could be billions of dollars of damage here. There must be billions in liability as well to balance the economic scales. This is way past the single individual who pulled the trigger.

Agree with the sentiment of not hiding behind vendors for issues, but in this case there were (presumably) major procedure failures at CrowdStrike which lead to this - I believe they are 100% at fault in this case.

> The hospitals, airlines, police, etc. that bought into CrowdStrike should have primary responsibility

This is ridiculous.

None of these business can (or should) handle all of their infosec internally. They must use vendors. And because they don't have infosec expertise, they have no way to verify that a vendor has good quality control.

You're essentially arguing that a hospital should also be responsible if people die because they bought the wrong MRI machine brand or one of their trucking suppliers had a crash and couldn't bring them enough of a certain drug.

There is no universe where vendors shouldn't be responsible for failing to deliver a safety-critical service that they 100% guaranteed they could deliver. There was nothing in the CrowdStrike contract that said, "Updates may inadvertently disable all systems that receive them," because that would mean no one should ever use it.

Hospitals shouldn’t use vendors like this. They need to be able to audit and understand what they’re doing and not allow. Hospitals have CIOs and CMIOs and CHIOs to make these kinds of decisions.

Computer security is clearly not the core competency of hospitals, nor should it need to be. I'd argue, putting Internet connected systems running MS Windows in the critical path was already setting them up for failure.

I would say every company of a certain size or criticality should have a proper IT team. It does not even need to be large, but they need to have the expertise and authority to veto certain decisions. I often see vendor deals being made solely for the discount and other marketing crap, like getting another piece of service free as a bonus. IT then gets silenced because number goes up. I have seen situations where leaderhsip keeps asking IT/Engineering the same question "But why can't we just do this?" in different ways and dissecting the answers, asking for reports, and proof, and what not, so that IT is forced to make a bulletproof case against it. It then appears that IT has a personal thing against the vendor. So IT gives up. System is up. Often, part of the problem is the vendor buying lunch to key decision makers and being pals.

So what should have told those CIOs not to use Crowdstrike? Who should they have used instead?

I am not familiar with how CS is or can be administered.

As a customer you should not have to assume the software your supplier vends is faulty to the extent of this incident.

It's reasonable for a customer to assume the software is tested according to industry best practices.

CS acted grossly negligent here, and they deserve the majority of the blame.

I agree that some blame also resides with customers, there must be disaster recovery procedures in place to allow them to function with minimal downtime in the case of emergency services.

I think the problem with DR in this case is you will need to account for every eventuality of every line of code in every sw product you use. Because say, how do you handle the case that your fleet BSODd? Usually you'd rebuild them using your bitlocker keys.

But this incident was so deep, that the SPOF here was using Windows. So now, your DR plan needs to account for some mandatory percentage of your OSs not being Windows, and your IT staff being maybe Linux experts. Cool.

But can you predict that you need to store your bitlocker keys in both platforms? And can you even do that or is it one of those things where bitlocker storage has to be on windows bc of lockin?

They should pay. And then the CrowdStrike and their shareholders should pay to them. We really should just simply remove corporate veil. Make the shareholders at least pay monetarily if not in prison time.

CRWD is a member of the S&P 500. Are you truly suggesting that every holder of the S&P 500 should be put in prison? Realistically, this means that nearly every American and a very large number of international average workers just trying to save for their retirement would have to go to jail. Sounds like a good way to inflate those private prison stocks.

See, that's the way the system has been crafted. Letting theives/incompetents get away with their mistakes/crimes is tightly coupled to people getting money for retirement.

Who's gonna rock that boat?

People are now literally invested in in a system that tolerates, if not ofttimes rewards, bad behavior from companies all the way from small things right up to "deliberately poisoning/cooking the whole ecosystem to make a buck"

Yeah calm down chief. You sound 13 and cool.

Very edgy.

I am a shareholder in dozens of companies. I have zero oversight on how things are managed. The only thing I can do is sell my paltry amount of shares if I think things are not going well there.

Shareholders would be punished in this case in the sense that crippling fines and reparations that Crowdstrike should definitely be liable to pay would decimate the stock value.

And if criminal charges were to be pressed, then the people actually in charge of the company should be on the hook. For example, if someone dies because CS was horribly negligent in their duties, then that should be a possibility.

Mr. Kurtz, is that you?

What does McKinsey has to do with it?

consultants spouting industry-standard slop, which means now everyone is on the same platform -- which we've just down doesn't really cover their butts.

and to be fair, having some sort of tool like that isn't necessarily a bad thing, the problem is when the tool gets pushed without testing and breaks the internet.

Yeah, why should software companies ever be held to some sort of standard? Let the buyer beware! Oh but don't call me a programmer, I'm totally a software engineer and don't you forget it.

I’ve never asked to be called an engineer. I guess I’m off the hook.

Reply on news.ycombinator.com