Seriously though - this entire outage is the poster child for why you NEVER have software that updates without explicit permission from a sysadmin. If I were in congress, I would make it illegal, it's an obvious national security issue.
Kernel level code blindly loading arbitrary files?
Panicking when the file doesn’t parse because it’s not a memory safe language?
Not validating the files before loading them?
Not validating the files before SHIPPING them? No CI? No safety net?
No staged rollout in case of explosion?
There are far FAR bigger mistakes here than “sys admin didn’t have to press button”.
This is a faulty and dangerous product from conception to execution.
Unfortunately "security" folk these days are box ticking fuckwits and this product brief ticked all the boxes. They do not understand any more traditional methodologies other than "install these magic beans and action the reports".
Invest in better software and network architecture and DR strategy instead.
Whether a program panics or recovers when attempting to parse bad data is entirely orthogonal to memory safety. Do you have any in-depth technical information about the bug itself that you're basing this on?
I agree with the rest, especially the use of a memory unsafe language to do parsing in the kernel by a billion dollar security company blows my mind.
How can you even run a security company without any security professionals reading your code even incidentally? An impressive level of incompetence.
I lean towards breaking things being the bigger risk.
But if even a handful of the other errors were corrected this would have been prevented and they wouldn’t have had to make that choice.
How the heck they didn't find out the new version prevent the computer from booting at all?
Who's going to take the risk of appearing to have sat on an important update, while the org they support is ravaged by ThreatOfTheDay, because they thought they knew better than a multi-billion dollar, tops-in-their-field company?
(I'm not necessarily saying that's actually objectively correct, but I can't imagine that many folks are willing to risk the downside)
In general I agree, but this case is quite messy. It's more like your anti-virus had a bug since forever that if it loads a broken virus definition it bricks your system. And a broken virus definition finally happened today.
Do you want every virus definition (that is updated every few hours) to require explicit permission from a sysadmin?
This example is probably an argument for not running windows on critical systems due to insufficient focus on security from the beginning which has lead to a need for things like crowdstrike.
They do make a version of CS for Linux but nobody runs it unless they’re forced to by overzealous compliance drones.
I wish people would stop making blanket statement as if they know how every company in the world runs. Plenty of Linux machines are running CS, and it's not only because they are forced to for compliance. NG AV has been picking up speed as a "just in case" thing for Linux and Mac for years now. Your anecdote does not apply to everyone.
The really important machines are still on Win 3.1.
I consulted for a company for a while where the 'sysadmin' was the owner's mother - who bought laptops from walmart. Not only could she NOT have approved updates like this, even if she could she would have she wouldn't have had any knowledge whatsoever with which to make a determination if it worked.
In an abstraction, the problem really is with externalities. These approaches to updates exist because people who CAN'T do what you describe are likely a more dominant part of the threat model than this happening to people you do describe. The resulting fix, as we're seeing, is very reliable until it isn't...and if the isn't is enormous in scale the systems aren't setup to fail gracefully.
If you want to make a rule...require graceful failure.
It's not like they'd read the source code or examine every file that's been changed or downloaded for a proprietary kernel module for every crowdstrike update (there must be a LOT of them).
They're the same team who mandate you use a 3-year-old browser version and 5-year-old OS, because you can't be trusted to manage your own updates, so they do know the idea.
Again, the same team in most enterprises wouldn't dream of letting you have an auto- updating Firefox Nightly, they know how to configure software so it doesn't phone home for updates or is blocked from phoning home.
If a code path isn't followed until a config file updates, that is practically the same thing as the code path being introduced by the update.