Hacker News Reader

Comment by easytiger | Hacker News Reader

Surely i'ts not normal practice to allow patches to be rolled out without a staging/testing area on an estate of that size?

View on HN

This is insane. The company I currently work for provides dinky forms for local cities and such, where the worst thing that could happen is that somebody will have to wait a day to get their license plates, and even we aren't this stupid.

I feel like people should have to go to jail for this level of negligence.

Which makes me think--are we sure this isn't malicious?

As strange as it sounds, this just seems way to sophisticated to be malicious.

Maybe someone tried to backdoor Crowdstrike and messed up some shell code? It would fit and at this point we can't rule it out, but there is also no good reason to believe it. I prefer to assume incompetence over maliciousness.

Unfortunately, any sufficiently advanced stupidity indistinguishable from malice.

The AI said it was ok to deploy

I blame the Copilot

True for all systems, but AV updates are exempt from such policies. When there is a 0day you want those updates landing everywhere asap.

Things like zscaler, cs, s1 are updating all the time, nearly everywhere they run.

>True for all systems, but AV updates are exempt from such policies. When there is a 0day you want those updates landing everywhere asap.

This is irrational. The risk of waiting for a few hours to test in a small environment before deploying a 0-day fix is marginal. If we assume the AV companies already spent their sweet time testing, surely most of the world can wait a few more hours on top of that.

Given this incident, it should be clear the downsides of deploying immediately at a global scale outweigh the benefits. The damage this incident caused might even be more than all the ransomware attacks combined. How long to take to do extra testing will depend on the specific organization, but I hope nobody will allow CrowdStrike trying to unilaterally impose a standard again.

It's incredibly bad practice, but it seems to be industry normal as we learned today.

I wonder if the move to hybrid estates (virtual + on prem + issued laptops etc) is the cause. Having worked in only on prem highly secure businesses no patches would be rolled out intra week without a testing cycle on a variety of hardware.

I consider it genuinely insane to allow direct updated from vendors like this on large estates. If you are behind a corporate firewall there is also a limit to the impact of discovered security flaws and thus reduced urgency in their dissemination anyway.

Most IT departments would not be patching all their servers or clients at the same time when Microsoft release updates. This is a pretty well followed standard practice.

For security software updates this is not a standard practice, I'm not even sure if you can configure a canary update group in these products? It is expected any updates are pushed ASAP.

For an issue like this though Crowdstrike should be catching it with their internal testing. It feels like a problem their customers should not have to worry about.

Their announcement (see Reddit for example) says it was a “content deployment” issue which could suggest it’s the AV definitions/whatever rather than the driver itself… so even if you had gradual rollout for drivers, it might not help!

It's definitely the driver itself if it blue screens the kernel. Quite possibility data-sensitive of course.

It might have been a long-present bug in the driver, yes, but today's failure was apparently caused by content/data update.

In most appreciations of risk around upgrades in environments with which i am familiar, changing config/static data etc counts as a systemic update and is controlled in the same way

You would lose a lot of the benefits of a system like crowdstrike if you waited to slowly roll out malware definitions and rules.

Survived this long without such convenience. anything worth protecting lives behind a firewall anyway

https://x.com/brody_n77/status/1814185935476863321 [0]

The driver can't gracefully handle invalid content - so you're kinda both right.

[0] brody_n77 is:

   Director of OverWatch,
   CrowdStrike Inc.

Looks like a great way to bypass crowd strike if I'm an adversary nation state

I came to HN hoping to find more technical info on the issue, and with hundreds of comments yours is the first I found with something of interest, so thanks! Too bad there's no way to upvote it to the top.

Anyone copy the original text? Now getting: > Hmm...this page doesn’t exist. Try searching for something else

I don’t have the exact copy, but it said it was a ‘channel file’ which was broken.

Reply on news.ycombinator.com