Wait, how is this stupid? Unless I'm missing something, wasn't the patch part of a Microsoft payload that included an update to Crowdstrike? Surely Crowdstrike is culpable, but that doesn't completely absolve Microsoft of any responsibility, as its their payload.
What is the point of driver signing if a vendor can basically build in a back door and Microsoft doesn't validate that this back door is at least somewhat reasonable
Backdoors of all kinds can be installed to most any operating system without vendor co-operation. That is the nature of general-purpose operating systems.
Perhaps you should push this change up in the food chain, then, and if the company is good the request will be taken seriously. As I understand it, while CrowdStrike is the biggest name in EDR, it's far from the only one, if that's what your company requires to pass some checkboxes in certifications.
CIO/CISO don't select vendors lightly.
There seems to be a typical/classical Engineer's mindset of "make a claim first, ask later" around the subject lately.
"My boss plays golf with Sales Rep" might need more proof because if they selected the lesser capable vendors and they got hit with ransomware, bet my ass your boss will no longer play Golf with any Sales Rep ever.
Sure, in a well-functioning market economy without any distortions. But there are lots of those at play, so competition is severely hampered (by network effects, regulatory capture, and on and on... Up to and including, I suspect, mere ephemeral fashion). What we actually have in many areas of the "tech market" are oligopolies and near-monopolies, not perfect competition.
> CIO/CISO don't select vendors lightly.
Muahaha. Seems rather more like they're at least as naïve as any Web-surfing consumer on their sofa, easily bamboozled by trendy buzzwords and slick marketing campaigns.
I've worked for a company that installs Falcon on all its fleet and I never run into issues like yours.
It would be one thing Microsoft could do to focus 100% of the attention/blame away from Windows and onto CloudStrike. And customers will want their pound of flesh from somewhere.
Really, this should serve as a wake up call w/in Microsoft to start to harden the kernel against such vulnerabilities.
Was the crash the fault of Windows? No. But did a Windows design decision make this possible? yes.
I’m sure the design decision made sense at the time (at least business sense). Keeping the kernel more open for others to add drivers to makes it easier to write/add drivers, but makes the system more vulnerable. This a good opportunity within Microsoft to get support for changing that.
If by changing the system you mean adding some kind of in-kernel isolation to it, then I don't think it would be worth the effort to make that kind of major change to the way operating systems work just to give arguably a minor risk reduction to systems—in particular if CrowdStrike and other vendors take some learnings from this event.
Microsoft might improve their system rollback mechanism to also include files that are not strictly integrated to the system, merely used by the parts that are (the channel files loaded by the driver).
Actually I think we can just be happy that the incident was a mistake, not an attack. Had this kind of "first ever" situation been an attack, it could be extremely difficult to recover from it. I wonder how well EDRs deal with "attacks from within"..
CrowdStrike pulled off the update within 1.5 hours. I wonder if they actually use Falcon themselves? But then somehow missed the problem? Doesn't seem like they eat their own dog food :). (Or at least their own channel files.)
I talked to my dad (retired enterprise operations/IT) this weekend and he was telling me that the next computer he buys will probably be a Mac, largely because he doesn’t want to deal with the possibility of a crash like this. Does he run CloudStrike? Not at all. Does he know who they are? Nope. (He’s been retired for a while) What he does know (well, thinks) is that Windows now has an unstable kernel.
And Microsoft has no control over distribution policies for other vendors. How those vendors distribute updates is up to them. Even if a sane deployment strategy could have avoided the larger global problems, Microsoft can’t control that.
So, if you have Microsoft dealing with negative publicity and public sentiment, with no way to control errors like this in the future, what can you do? To me, the best they can do is kneecap CloudStrike, put the full blame on them, and use this as an excuse to change the kernel/driver model to one where they can have more control over the stability of the OS.
Microsoft could even reinstate CloudStrike at some point, but only after an extensive review process. And then probably require similar process reviews/checks for any other vendor that requires the same kernel access.
Or just remove the need for kernel access at all and migrate to a better driver architecture at the sacrifice of backwards compatibility. Security software doesn’t need to run in kernel space… there are other ways.
macOS has a userland API for this, called EndpointSecurity, which allows doing all the things an EDR needs, without ever touching kernelland. Microsoft could introduce a similar API, and EDRs would no longer need a driver.
It would certainly help solving this particular problem, even if not the kernel-integration in general.
I’m sure the reason why Apple migrated was because of all of the bugs/crashes security companies kept introducing into the kernel with kexts. Apple had the ability to change their architecture on a whim because of they aren’t quite a beholden to backwards compatibility as Windows.
Microsoft could take this as an opportunity to make some major changes that would be more readily accepted by the market.
Not on Kernel level. Not without active support by the vendor.
Actually I could easily see a buggy remote system management update could just decide to uninstall everything and nuke the system, because it thinks it's stolen. And it would be designed functionality for it.
From my understanding the CS driver lives in the kernel space and parses configs/applications downloaded in the user space. Hence the system even does a BSOD.
"CrowdStrike doesn't have kernel-level ACE" please provide your source.
That's my understanding as well, but not quite the same as
> execute arbitrary code on the kernel level without any checks
At least for me, when we talk about kernel-level ACE, it's something like libcapcom[0], which allowed executing arbitrary unsigned code in the kernel.
Here, the driver can only execute the code present within itself, which was signed by Microsoft. The configuration itself isn't signed by microsoft, but the config isn't code (at least, as far as I can tell - I see some people claiming the CS-0000.sys files are essentially bytecode, but have yet to see conclusive proof of this).
Now, we could argue that it's weird that Microsoft signed a buggy driver, and MS should do better qualification of third-party drivers. But in practice, MS doesn't really vet driver quality. From what I can tell, the driver signing is mostly there so they can easily attribute provenance of drivers, and revoke the certs if it ends up in the hands of malicious actors.
but AFIK no sources I trust have yet claimed it
but it's probably where the idea comes from
That would be grounds for blacklisting indeed (out of experience). However, that's not the case here, no matter how you put it.
Before you downvote that comment, I'd like to remind everyone that this was already happening. Realtek's driver cert was leaked, and a lot of malware used this cert to sign their drivers for _a decade_ until anything happened about it.
Microsoft's driver signing workflow is utterly pointless and it doesn't mean anything. Any vendor that takes their security serious should never trust those driver signatures.
Maybe I'm wrong and Microsoft does some QA on drivers before they are signed?
this comment right here sums it up:
> Sure, but Windows shares some portion of the blame for allowing third-party security vendors to “shit in the kernel”.
On the other hand, Linux is similarly crippled: eBPF LSM are fairly recent and don't work everywhere (I'm looking at you Ubuntu[0]), and the only real alternative if you want to be able to block processes is a kernel module. Which comes with the same dangers as Windows.
[0]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2054810
I have my doubts, but that's at least what they give as the reason for the kernel-level access of EDR tools.
This update was buggy and it caused the host machine to go into a BSOD boot loop.
The fact that the host machine happened to be running Windows has very little to nothing to do with it.
It's like blaming a pothole for your car going exploding. Yes there was a pothole, yes it shouldn't have been there, yes it could have been avoided, but the fact that your car self immolated because of mere pothole points to possibly other underlying issues with your car.
Maybe most of these "drivers" shouldn't be in Ring0 to begin with? This is a general problem and the norm, Windows is just another OS that allows this this way.
user-land drivers are a thing, heck they are the standard for modern micro kernel architectures
and even with hybrid kernels pushing part of the code out of the kernel into something like "user land co-processes" is more then doable, now it's now trivial to retrofit in a performant way and flexible way but possible
Mac has somewhat done that (but I don't know the details).
On Linux it's also possible, through with BPF a bit of a in-between hybrid (leaving some parts of the drivers in kernel, but as BPF programs which are much less likely to cause such issues compared to "normal" drivers).
A good example for that is how graphic drivers have developed on Linux, with most code now being in a) the user-land part of the driver and b) on the GPU itself leaving the in kernel part to be mostly just memory management.
And the thing is Windows has not enforced such direction, or even pushed hard for it AFIK, and that is something you can very well blame then for. You in general should not have a complicated config file parser in a kernel driver, that's just a terrible idea, some would say negligent and Windows shouldn't have certified drivers like that. (But then given that CrowdStrike insists that it _must_ be loaded on start (outside of recovery mode) I guess it would still have hung all systems even if the parsing would have been outsourced because it can't start if it can't parse it's config).
Even here it's pretty hard to blame them due to antitrust concerns. Just google the word Patchguard.
Falcon uses apis like eBPF when available/usable , they are not stupid if they can use something which is more secure and reliable why should they not use it.
E.g. they use it on Linux, even through they could have created a custom kernel module (idk. if they maybe also have a custom kernel module tbh.).
And pushing for something doesn't mean banning other things. E.g. they could certify "following best security practices" and not give it to vendors not using the more modern APIs, while they can't block drivers based on it with the right marketing customers of CrowdStrike wouldn't want to buy it without such cert.
I.e. while MS doesn't provide viable ways to get the functionality Falcon and similar need without kernel modules it indeed would be a bit ridiculous for them to ban such software, and as of yet the do not.
Where Microsoft is to blame however is the unrelated Azure outage in the Central US region that happened (and was fixed) just before the CrowdStrike faulty update.