Hacker News Reader

Comment by retrochameleon | Hacker News Reader

CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments. The bug that caused all this was present in their kernel driver for a long time.

For being such a large cybersecurity player and deploying updates to 8.5 million devices, their quality control practices are embarrasingly lacking.

View on HN

> CrowdStrike blamed their test software, but in the same breath revealed that they haven't been using any canary deployments.

Their post-incident report [1] also stated that they intend to improve testing by "using testing types such as: local developer testing". One has to wonder what, if any, testing they were doing beforehand.

[1]: https://www.crowdstrike.com/blog/falcon-content-update-preli...

Well we know what the testing is, don’t we?

The update literally crashed the system it was used on.

There’s no way they couldn’t know that unless they never ran it. Right?

Is this one of those things that only happened to 10% of users? Because I haven’t seen that reported anywhere.

> Is this one of those things that only happened to 10% of users? Because I haven’t seen that reported anywhere.

As far as I'm aware, it affected all systems using Crowdstrike.

Every company I've ever been at rolls out updates slowly. Rolling out a change to 8.5 million computers at the same time seems ridiculous. Even the most cash strapped start-ups with every incentive to cut corners tends to get staged roll-outs more or less right. It's crazy.

Beyond crazy. I even have a small app that never makes it to production before being rolled out to internal and open testing first. And, even then, it's slowly rolled out to a percentage at each stage before being fully deployed. One would think a major company with kernel level access would do this at minimum.

I had a fleet of only maybe 200 computers I updated remotely. I did canary staged roll outs.

When I managed ~ 15 developer’s Arch Linux workstations, I found it very beneficial to be the canary, and then rollout to a couple of the more capable of troubleshooting devs, and then the rest. I can always fix my own box.

8.5M all at once feels insane.

not a software update!

Not relevant!

details are always relevant in a technical discussion. look at my other comments where i pointed out microsoft performing similar immediate av signature updates and causing chaos.

Some details are relevant, some are not.

I'm more than comfortable labelling parts of Microsoft as incompetent as well.

We can agree on that, but it is relevant because this isn't an unusual practice. Crowdstrike didn't ignore some pre-existing best practice. Lots of things need improving but facts and details matter when you talk about RCA. it isn't about blame but fixing the root cause.

Deploying an update to 8.5 million devices without doing any form of staged roll-outs, not even just updating a few machines in the office first, is absolutely bonkers. If that's not an unusual practice it bloody should be. I don't care if it's a software update or a config update.

again, this is why I was snarky in my earlier post, this was not a software update. they should have used canary deployments still but in many cases prior to this incident, it was not acceptable to wait even a few hours because it can make the difference between companies getting ransomwared/hacked, so they focused on making the actual code/driver that interprets the channel file updates robust enough to handle real-time updates. Even if other players were doing canary deployments with behavioral detection updates, they're not the market leader, crowdstrike is for a reason.

Everyone that worked in an operational incident response role has blocked some indicator like an ip address or a domain. you don't do gradual roll outs for those either, and i've seen people cause outages by skipping a check or making a mistake. this is similar in many ways to that except it was for a named pipe. This could probably have waited for a canary deployment, but in general the class of content that is being deployed would be deployed right away, I'd be surprised if their practice is considered "bad" by any measure. I've seen Microsoft also deploy email quarantine signatures and defender updates that caused large scale impacts.

Here is a link of what Microsoft did earlier this year:

https://www.techradar.com/news/google-chrome-not-working-mic...

If they had canary deployments, that wouldn't have happened. I had rules that were causing chaos because of that. Now imagine if defender had a bug that caused it to crash because of a signature update. The impact would be magnitudes greater than what you saw with Crowdstrike. It's really frustrating to see the lack of technical critical thinking and arm-chair experts acting like they know what they're talking about.

> it was not acceptable to wait even a few hours

Hours... Wouldn't a 15 minute canary have found this problem about 14 minutes before it hit wider deployment?

Let's say the driver was "robust enough" to handle a broken channel file. How would that look exactly? Say you're responsible for writing the code which loads a new channel file. These channel files are critical; without them, your security critical product doesn't know how to do its job. The channel file parser returns a parse error. How should the driver respond? Surely you're not going to just silently disable your security critical product if someone puts a bad channel file in there?

Delete the file or mark it as corrupt so that the parser doesn't keep trying to read it, and send some telemetry back to CS to indicate there is a problem with the one of the channel files. It doesn't seem very complicated at all. There are plenty of options in between "catastrophically crash the OS" and "silently disable the entire product".

That seems pretty dangerous if that channel file included security critical configuration, which it presumably did

Clearly incompetence to deploy from 0 to 8 million devices without any gradual rollout.

That goes even further, because apparently they were fully blind and didn't have crash metrics.

"Ok we push the update, and pray".

I think it is past incompetence, and on into negligence. Given the stories we have heard here about emergency service failures it is likely that people died. When people die due to negligence isn't that usually criminal?

Can't agree more, you found the right words.

And this is how the lawsuits will start.

Who is negligent though? Crowdstrike, or the emergency services that are using an OS that requires third party endpoint security right out of the box in order to be safely used, or the company that makes and sells that OS?

Why not both?

Crowdstrike, for negligently not rolling out updates gradually.

And emergency services, if they don't have robust fallback procedures/systems for when their IT system goes down. I mean it's totally fine if regular doctor's visits get postponed, but 911 should never go down just because their computers down. Just like aircraft have redundant systems, so too should 911.

(The company that makes and sells the OS -- I don't see any negligence there, in this case. If security software fundamentally requires running at the kernel level and Microsoft allows that, I don't see how Microsoft can be at fault.)

Yeah, I don’t see how one can blame Microsoft in this scenario. If you choose to run buggy kernel-level code, that’s on you, not the publisher of the kernel/OS. Especially when the code you’re running is a replacement for functionality already provided by the OS. It’s hard to argue that MS could be negligent for “not having a good enough AV/endpoint protection solution” or “allowing customers to run kernel-level code.”

It’s hard for people to understand that these massive ‘security’ enterprises are often connected by a large amount of bodies instead of competence.

https://www.techradar.com/news/google-chrome-not-working-mic... ,not an unusual practice and they were not first av company to cause outages. and again, it was not a software update, the buggy software was deployed after testing back in march. Details matter!

How about we let the lawyers figure out who had what liability, just like with the av/edr industry, we should know when the subject matter is outside our area of knowledge and expertise.

What isn't an unusual practice? A lack of canary deployments? For av definition files, where crashing the scanner doesn't bring down the kernel I can agree. For a file read by a kernel module/driver that is primarily used by security sensitive infrastructure, I really hope it is unusual, and if it isn't lets make an example.

If failing to do unit tests to get full coverage (line and branch) on software that gets loaded on security sensitive infrastructure isn't unusual then things have sure changed in the decade since I was involved. We had to get that coverage at the assembly level and then peer review the tests to make sure they were meaningful and correct. Every time the code changed. Code that failed to properly sanitize input files without panicking should not pass normal safety critical code pipelines.

I do wonder if windows already has some disclaimer about being unfit for such use cases, in which case yeah, lawyers will have some fun, but even with that sort of protection, CS is clearly marketing to critical infrastructure, so it has no moral defense whatever lawyers and the law says.

I shared with a sibling commenter:

https://www.techradar.com/news/google-chrome-not-working-mic...

Did Microsoft do a staged or canary roll out with that? This is not a software update, if you're making such comments then you're speaking about something outside of your field of expertise.

Unless their developers had room temperature IQs or were actual psychopaths, I really wonder how they even managed to find developers who had the nerves to deploy to the whole world all at once like that. If it were me I'd be scared shitless, covered in sweat and probably shaking too hard to even type. Were CrowdStrike developers too stupid to even realize the magnitude of what they were doing? Or did they have cooler nerves than an open-heart surgeon? It's shocking to me that they could have done this so casually.

> Were CrowdStrike developers too stupid to even realize the magnitude of what they were doing?

More likely they were following a playbook to the letter, and were therefore 100% of success.

Reply on news.ycombinator.com