Is that because of economies of scale or because the vendor is just cutting costs while hiding their negligence?
I don't understand how a single vendor was able to deploy an update to all of these systems virtually simultaneously, and _that_ wasn't identified as a risk. This smells of mindless box checking rather than sincere risk assessment and security auditing.
That's how it works.
The whole market in which crowdstrike can exist is a result of regulation, albeit bad regulation.
And since the returns of selling endpoint protection are increasing with volume, the market can, over time, only be an oligopoly or monopoly.
It is a screwed market with artificially increased demand.
Also the outsourcing is not only about cost and compliance. There is at least a third force. In a situation like this, no CTO who bought crowdstrike products will be blamed. He did what was considered best industry practice (box ticking approach to security). From their perspective it is risk mitigation.
In theory, since most of the security incidents (not this one) involve the loss of personal customer data, if end customers would be willing to a pay a premium for proper handling of their data, AND if firms that don’t outsource and instead pay for competent administrators within their hierarchy had a means of signaling that, the equilibrium could be pushed to where you would like it to be.
Those are two very questionable ifs.
Also how do you recognise a competent administrator (even IT companies have problems with that), and how many are available in your area (you want them to live in the vicinity) even if you are willing to pay them like the most senior devs?
If you want to regulate the problem away, a lot of influencing factors have to be considered.
I know you aren't saying it is, but I think Taleb would argue that this incident, as he did with the coronavirus pandemic for example, isn't even a Black Swan event. It was extremely easy to predict, and you had a large number of experts warning people about it for years but being ignored. A Black Swan is unpredictable and unexpected, not something totally predictable that you decided not to prepare for anyways.
Apart from a few doomsdayers I am not aware of anybody was warning us about a crowd strike type of event. I do not know much about public health but it was my understanding that there were playbooks for an epidemic.
Even if we had a proper playbook (and we likely do), the failure is so distributed that one would need a lot of books and a lot of incident commanders to fix the problem. We are dead in the water.
I don't think centrally distributed anti-virus software is the only way to maintain reliability. Instead, I'd say companies to centralize anything like administration since it's cost effective and because they actually aren't concerned about global outage like this.
JM Keynes said "A ‘sound’ banker, alas! is not one who foresees danger and avoids it, but one who, when he is ruined, is ruined in a conventional and orthodox way along with his fellows, so that no one can really blame him." and the same goes for corporate IT.
In all the cases I know, we traded frequent and localized failure for infrequent but globalized catastrophic failures. Like in this case.
A system that can tolerate anything, so have perfect availability, seems clearly impossible. So yeah, totally right, it's always a tradeoff. That's reasonable, as long as you trade smart.
I wonder if the people deciding to install Crowdstrike are aware of this. If they traded intentionally, and this is something they accepted, I guess it's fine. If not... I further wonder if they will change anything in the aftermath.
- This is system has a single point of failure, it is not fault tolerant. Lets introduce these three things to make it fault-tolerant
- Now you have three single points of failure......right?
tldr there are levels to fault tolerance and I understand what you are saying. I am not sure if you are advocating for getting rid of fault handling, but generally you can mitigate the big scary monsters and what is left is the really edge case issues, and there really is no stopping one of those from time to time given we live in a world where anything can happen at anytime.
This instance really seems like a human related error around deployment standards...and humans will always make mistakes.
So yeah, there is no fault tolerance if the timeframe is large enough, there are just less events, with much higher costs. It's a tradeoff.
The cynical in me thinks that the one advantage of these complex CYA systems is that when systems fail catastrophically like CrowdStrike did, we can all "outsource" the blame to them.
> Why must everything collapse? Because, [Kaczynski] says, natural-selection-like competition only works when competing entities have scales of transport and talk that are much less than the scale of the entire system within which they compete. That is, things can work fine when bacteria who each move and talk across only meters compete across an entire planet. The failure of one bacteria doesn’t then threaten the planet. But when competing systems become complex and coupled on global scales, then there are always only a few such systems that matter, and breakdowns often have global scopes.
https://www.overcomingbias.com/p/kaczynskis-collapse-theoryh...
There was a quote last year during the "Twitter files" hearing, something like, "it is axiomatic that the government cannot do indirectly what it is prohibited from doing directly".
Perhaps ironically, I had a difficult time using Google to find the exact wording of the quote or its source. The only verbatim result was from a NYPost article about the hearing.
Turns out SCOTUS decided it isn't, and the government is free to do exactly that as long as they are using the services of an intermediary.
Sorry. Not true. You have Black Swan (Taleb) and Drift into Failure (Dekker) among many other books. These ideas are very well known to anyone who makes the effort.
Be realistic, none his ideas would be blacklisted. They sound good on paper, but the instant it's time for everyone to return to mudhuts and farming, 99% of people will return to Playstations and ACs.
He wasn't "silenced" because the government was out to get him, no one talks about his ideas because they are just bad. Most people will give up on ecofascism once you tell them that you won't be able to eat strawberries out of season.
Hell, if we take his alleged (don't have vetted the genealogy tree) cousins, his body count isn't even that impressive.
I suspect most people in power just don't subscribe to that. which is precisely why it's systemic to see the engineer shouting "no!" when John CEO says "we're doing it anyway." I'm not sure this is something you can just teach, because the audience definitely has reservations about adopting it.
You can't prevent failure. You can only mitigate the impact. Biology has pretty good answers as to how to achieve this without having to increase complexity as a result, in fact, it often shows that simpler systems increase resilliency.
Something we used to understand until OS vendors became publicly traded companies and "important to national security" somehow.
https://simons.berkeley.edu/events/lessons-texas-covid-19-73...
The only possible way to fault tolerancy is simplicity and then more simplicity.
Things like crowsdtrike have the opposite approach. Add a lot of fragile complexity attempting to catch problems, but introducing more attack surfaces than they can remove. This will never succeed.
Basically delegation.