CrowdStrike CEO summoned to explain epic fail to US Homeland Security committee
theregister.com
theregister.com
I guarantee they all had BC/DR plans.
I honestly believe the whole thing in general has net negative ROI just because it never works.
What's absurd about it? It's absurd that I want the software operating my MRI and providing doctors with necessary information during surgeries to be different from what accountants using Excel and video game players use?
What's absurd is congress hoping to solve the fundamental problem by asking the CrowdStrike CEO to come in for a one on one. Even if we took this process at face value and assumed they were earnest it would be laughable.
Meanwhile.. the government is the largest healthcare payer in the world. For not particularly great reasons. This _is_ one of the downstream impacts of that reality.
This has nothing to do with there being a healthy market of operating systems to choose from.
> I’ve seen the fragmentation in edge devices
How many versions and patch levels of Windows exist? How many are currently deployed? How many must stay on the current version because of fragmentation?
> You know how many bugs you have with one stack?
The scenario that started this discussion is a single bug in a single piece of software impacting a large number of businesses, some of which are safety and life critical, all of which are on a single OS.
It's not a question of "how many bugs," because you can't fully eliminate them in any system which is actually updated _ever_, it's a question of "how many people does this single bug impact?"
Will this require more time and labor? Yes, of course. Does that make it "absurd" to consider? No, that's pure hyperbole.
If the website fails, that costs a lot of money, but nobody is in danger. If dispatch systems fail that has a safety risk to flight ops.
While you may not be happy that your flight gets cancelled. For airline IT there is a cost vs benefit balance between the probability of flight cancellations vs the cost of having extra systems. (And in this case extra systems would probably have been affected by the same security platform, because not having that is another risk of cyber attack to consider)
That doesn't apply to flight ops, where each operator needs to prove to a very high level that safety is always maintained. Maximum probability for catastrophic failure is 10E-9 and for hazardous is 10E-7. No commercial considerations considered, have to make those numbers to be allowed to operate.
Then again with systems working Boeing failed that one...
Except for massive disruptions to tons of people, businesses, and economies worldwide, with impacts rippling out for days after services have been restored?
1. https://www.npr.org/2024/07/23/nx-s1-5049792/deltas-airlines...
Should hospitals plan on not being able to provide medical care and not able to transfer their patients elsewhere, or divert the inflow of new patients elsewhere, because every other hospital is down too, there is no one left to provide care, and 911 is shot anyway? Maybe. Right after planning for a nuclear explosion over the city.
What happened was pretty much an "act of God"-level crisis, except caused by a de-facto infrastructure provider with way more destructive power than they're equipped to handle. IMO, not only CrowdStrike should be liable for the damage and lives lost because of the outage, this incident should be a prompt to rethink the entire idea of endpoint security.
--
[0] - I think it should be, but then I think the entire business with endpoint protection is bullshit.
No, but cyberattack takes out all computer infrastructure is a high impact, low risk that critical infrastructure providers must account for and would've prevented any impact from CrowdStrike's blunder. They obviously didn't do that.
Yes.
That decision maker was promoted and since left for a new leadership position.
Because we wont let you have your pet dev box naked, sorry.
Usually both are out of sight and out of mind. Curious what problems you’re having
Also, you probably don’t want to be responsible for causing your company to get ransomwared or data breached, which is the primary reason you won’t get any sympathy or special treatment.
/s
Who's gonna rock that boat?
People are now literally invested in in a system that tolerates, if not ofttimes rewards, bad behavior from companies all the way from small things right up to "deliberately poisoning/cooking the whole ecosystem to make a buck"
I am a shareholder in dozens of companies. I have zero oversight on how things are managed. The only thing I can do is sell my paltry amount of shares if I think things are not going well there.
Shareholders would be punished in this case in the sense that crippling fines and reparations that Crowdstrike should definitely be liable to pay would decimate the stock value.
And if criminal charges were to be pressed, then the people actually in charge of the company should be on the hook. For example, if someone dies because CS was horribly negligent in their duties, then that should be a possibility.
I wouldn’t want to live in a world where people can get away with mistakes like this: as far as I have seen, if one person in CS had actually tried this file on their local windows machine, it would’ve crashed. That’s not a small thing is it? That’s a Boeing like culture which should be terminated today.
I accept shared responsibility when there is some agreed deployment and such scenarios, but here? Nope it is on CrowdStrike...
If I give root to a third party, then I’m liable for everything they do with that root.
The correct comparison would be handing your house keys to a licensed insect exterminator and the guy accidentally poisoning your family. You of course are not liable if a professional supplying you with a service messes up.
It’s like paying a licensed insect exterminator that uses random chemicals. They tell you this and you hand over the keys anyways.
I don't consider them capable of preventing something like this from happening again.
I think trading, say, small x (15 minutes, an hour?) time, to stage rather than blast might be an acceptable tradeoff given the visible consequences of not doing so.
Our essential services should be built on a more secure OS, if windows isn’t suitable.
So it sounds like you agree that Crowdstrike should go bankrupt.
> Our essential services should be built on a more secure OS, if windows isn’t suitable
Oh wait you blame Windows for this?
- https://www.crowdstrike.com/blog/crowdstrike-brings-xdr-to-z...
Or maybe not to boot . . .
Because critical infrastructure doesn't get the convenience of shifting the blame. They must account for these risks in advance. Do you know what happens when IT breaks down in a bank? They've got two or more printers that print all new transactions. Database corruption? No problem, we've got the transactions right there. Printer breaks down? No problem, we've got another one printing the transactions right there. Hospitals don't get the luxury of shifting the cause of death from them failing to properly account for risks to a piece of software.
Hypothetically, if there was a widespread 'bug' in... Intel CPUs that caused a similar issue, would you say it was the hospitals that should be held accountable, or Intel, or someone else? What about if the issue was in Windows itself - what should each hospital do differently that would have avoided this?
When the damages are in the billions we need to make sure that there are billions in liability to balance the scales. Or else there will always be a perverse incentives to not give a shit.
Whose incompetence is that? If a hospital pays for an energy redundancy solution where the vendor basically communicates "please don't use us for anything important", and then a power outage costs lives, are we seriously saying that the hospital shouldn't have to pay big time?
I'm not saying that CS shouldn't be liable. I'm saying that when consumers are looking for someone to sue, and regulators are looking for someone to punish, those who directly interface with the consumer should be in the direct line of fire. Hospitals in turn should be able to sue CS.
We certainly should not prefer that the blame goes straight to the engineer as then the consumer would basically have nobody to sue. That's a world of even less accountability. Those who are most capable of taking responsibility should take responsibility, and not merely those who are most blameworthy from first principles of local causality.
There could be billions of dollars of damage here. There must be billions in liability as well to balance the economic scales. This is way past the single individual who pulled the trigger.
This is ridiculous.
None of these business can (or should) handle all of their infosec internally. They must use vendors. And because they don't have infosec expertise, they have no way to verify that a vendor has good quality control.
You're essentially arguing that a hospital should also be responsible if people die because they bought the wrong MRI machine brand or one of their trucking suppliers had a crash and couldn't bring them enough of a certain drug.
There is no universe where vendors shouldn't be responsible for failing to deliver a safety-critical service that they 100% guaranteed they could deliver. There was nothing in the CrowdStrike contract that said, "Updates may inadvertently disable all systems that receive them," because that would mean no one should ever use it.
As a customer you should not have to assume the software your supplier vends is faulty to the extent of this incident.
It's reasonable for a customer to assume the software is tested according to industry best practices.
CS acted grossly negligent here, and they deserve the majority of the blame.
I agree that some blame also resides with customers, there must be disaster recovery procedures in place to allow them to function with minimal downtime in the case of emergency services.
But this incident was so deep, that the SPOF here was using Windows. So now, your DR plan needs to account for some mandatory percentage of your OSs not being Windows, and your IT staff being maybe Linux experts. Cool.
But can you predict that you need to store your bitlocker keys in both platforms? And can you even do that or is it one of those things where bitlocker storage has to be on windows bc of lockin?
and to be fair, having some sort of tool like that isn't necessarily a bad thing, the problem is when the tool gets pushed without testing and breaks the internet.
“Is that a dot-matrix printer?” I asked.
“Yep. This system here—” and she tapped the monitor “— runs DOS.”
“Any downtime on Friday?”
“Nope. Solid as a rock.”
And I wonder do we really need all this complexity to run our every day stuff? I mean, yeah if you're running Photoshop or something, but basically moving numbers and demographics through a system, what the heck do you really need an ad-riddled privacy-invading behemoth like Microsoft Windows for?
So IMHO isn't it time to rethink this idea of putting every single shit into the kernel space?
In other words Linus won then, and its now time to think about Tanenbaum too.[0]
[0] https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
I wonder if it would be possible to make an OS that uses an inter-process communication primitive that works like io_uring to mitigate this overhead.
Why that is the case when we have faster CPU, RAM, and SSDs etc.
I think a big question is why are things like CrowdStrike still written in raw C++ Kernel code? Is it the limitations of eBPF? Is it stagnations of the tech stack at these companies?
Vendors must provide secure products and transparent communication, while customers need to make informed decisions and properly maintain the security measures provided.
In this case though, the vendor needs to clearly explain what happened.
As for the customers, they now need to analyze if they need to continue using the said vendor.
On a final note : never underestimate the power of mistakes - plain and simple. They exist and happen most usually.
There should never be implicit trust on critical infrastructure, period.
Seriously, who would run an operating system that requires trusting code by giving it ambient authority? We've known that was nuts since the 1970s.
> We hope he's not flying Delta to Washington DC, as the US airline has canceled more than 5,000 flights since Friday as a result of CrowdStrike's update file crashing Windows systems.
I thought I was taking crazy pills when I read that and started wondering where the world was headed. Im glad atleast the CEOs are expected to give answers.
If you are working for public institutions, then it probably matters immensely. But if you answer only to shareholders, then its more theatre.
Reform of companies like BlackRock is sorely needed, the way they operate is detrimental to the actual owners of equity, and society at large.
Furthermore, most of the money managed by BlackRock is not managed by them through direct choice of the equity holders, so even if the equity holders were not happy with BlackRock it would be very difficult for them to move their money elsewhere.
And then finally, the shares of BlackRock itself are also held on behalf of others by institutions like BlackRock, and some of it even by BlackRock itself. So they vote with the voting rights of others as to how they think they should be run.
The professional managerial class generally does not care about profits, as they exceed at finding ways of enriching themselves without having to actually deliver anything. They get rich while the average Joe's pension underperforms. They love things like DEI and ESG, as that allows them to define their own parameters for success in ways which in no way benefit the people they have a fiduciary duty to or society.
Thanks for your informative response
Technically, if your investment does well, they have more assets under managemnt, and they earn a larger fee, but in practice they can just increase their rate because the people using them aren't shopping around anyway, and there will always be more money invested through them.
In reality, most people are paying BlackRock without ever knowing that they are paying them, and then in addition to paying BlackRock in money they get the benefit of your voting rights, which they can then do with as they see fit. They are power brokers who are paid by the people whose power they are brokering.
- 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993
It seems to me that there's a need for the Crowdstrike CEO to take responsibility, e.g. for lax testing and for not doing canary rollouts.
And it seems to me that there's a place for the [airline|bank|hospital] CEOs and CIOs to consider implementing canary 'roll-ins' where new software is tried on a small scale before being installed everywhere.