Hacker News Reader

Would any of these, or even a collection of these, resolving in some direction make it highly improbable that it'll never happen again?

Seems to me 3rd party code, running in the kernel, on parsed inputs, that can be remotely updated is enough to be disaster waiting to happen gestures breezily at Friday

That's, in the Taleb parlance, a Fat Tony argument, but barring it being a cosmic ray causing a uncorrected bit flop during deploy, I don't think there's room to call it anything but "a disaster waiting to happen"

Yes, if CrowdStrike was following industry best practices and this happened, it would teach us something novel about industry practices that we could learn from and use to reduce the risk of a similar scale outage happening again.

If they weren't following these practices, this is kind of a boring incident with not much to be learned, despite how dramatic the scale is. Practices like staged rollout of changes exist precisely because we've learned these lessons before.

Well, kernel code is kernel code, and kernel code in general takes input from outside the kernel. An audio driver takes audio data, a video driver might take drawing instructions, a file system interacts with files, etc. Microsoft, and others, have been releasing kernel code since forever and for the most part, not crashlooping their entire install base.

My Tesla remote updates ... hmph.

It doesn't feel like this is inherently impossible. It feels more like not enough design/process to mitigate the risks.

kernel driver could have data check on the channel file and fail gracefully/ignore wrong file instead of BSOD.

this code is executed only once during the driver initialization, so shouldn't be much overhead, but will greatly improve reliability against broken channel file

This is going to code as radical, but I always assumed it was derivable from bog-standard first principles that would fit in any economics class I sat in for my 40 credits:

the natural cost of these bits we sell is zero, so in the long run, if the bar is "just write a good & tested kernel driver", there will always be one more subsequent market entrant who will go too cheap on engineering. Then, they touch the hot wire and burn down the establishment.

That doesn't mean capitalism bad, but it does mean I expect only Microsoft is capable of writing and maintaining this type of software in the long run.

Ex. The dentist and dental hygienist were asking me who was attacking Microsoft on Friday, and they were not going to get through to the the subtleties of 3rd kernel driver release gating strategy.

MS has a very strong incentive to fix this. I don't know how they will. But I love when incentives align and assume they always will, in the long run.

To answer some of my questions based on the "Preliminary Post Incident Review", the config file was indeed invalid, it was only checked with a (buggy) validator, and then released to the whole world at once. Never was the config file ever tested with the actual software that would read it in an actual environment like the customer machines that got this update.

They don't say why it was invalid or really what the file is, but it seems like it is some kind of relatively complex set of rules that are evaluated by the kernel module. Presumably they are manually authored and reviewed and it seems possible the bug was missed in review because this was a relatively new type of rule.

So this isn't a case of an incident that slipped through a rigorous testing and release process process following industry best practice, but rather a "disaster waiting to happen". Further, CrowdStrike CEO George Kurtz should have known better, considering an analogous incident happened under his watch as CTO of McAfee in 2010.

https://www.crowdstrike.com/falcon-content-update-remediatio...

The glaring question is how and why it was rolled out everywhere all at once?

Many corporations have pretty strict rules on system update scheduling so as to ensure business continuity in case of situations like this but all of those were completely circumvented and we had fully synchronised global failure. It really does not seem like business as usual situation.

The glaring question is how and why it was rolled out everywhere all at once?

Because the point of these updates is to be rolled out quickly and globally. It wasn't a system/driver update, but a data file update: think antivirus signature file. (Yes, I know it can get complicated, and that AV signatures can be dynamic... not the point here.)

Why those data updates skipped validity testing at the source is another question, and one that CrowdStrike better be prepared to answer; but the tempo of redistribution can't be changed.

A customer should be able to test an update, whether a signature file or literally any kind of update, before rolling it out to production systems. Anything else is madness. Being "vulnerable" for an extra few hours carries less risk than auto-updates (of any kind) on production systems. As we've seen here. If you can point to hard evidence to the contrary, where many companies were saved just in time because of a signature update and would have been exploited if they'd waited a few hours, I'd love to read about it. It would have to have happened on a rather large scale for all of the instances combined to have had a larger positive impact than this single instance.

But is there a need for quick global releases?

Is it realistic that there's a threat actor that will be attacking every computer on the whole planet at once?

I can understand that it's most practical to update everyone when pushing an update to protect a few actively under attack but I can also imagine policies where that isn't how it's done, while still getting urgent updates to those under attack.

Is there a need? Maybe, possibly, depends on circumstances.

Is this what people are paying CS for? Absolutely.

After this I imagine there will be an option "do you want updates immediately, or updates when released - n, or n+2, n+6, n+24, n+48 hrs?"

Given the choice I bet there's going to be surprisingly large number of orgs go "we'll take n+24hrs thanks"

> strict rules on system update scheduling

which crowdstrike gets to bypass because they claime themselves as an antivirus and malware detection platform - at least, this is what the executives they've wined and dined into the purchase contracts have been told. The update schedule is independently controlled by crowdstrike, rather than by a system admin i believe.

From the article on The Verge it seems that this kind of update is downloaded automatically even if you disable automatic updates. So those users who took this kind of issue seriously would have thought that everything was configured correctly to not automatically update.

CrowdStrike's reasoning is that an instantaneous global rollout helps them protect against rapidly spreading malware.

However, I doubt they need an instantaneous rollout for every deployment.

Well, millions of PCs bluescreening at the same time does help stop a rapidly spreading malware.

Only this time, crowdstrike itself has become indistinguishable from malware.

Whe I first saw news about the outage I was wondering what this malware "CrowdStrike" was. I mean, the name kind of sounds hostile.

They say that, but all I hear is immune system triggering a cytokine storm and killing you because it was worried you may catch a cold.

I feel like they need to at least first rollout to themselves

> The glaring question is how and why it was rolled out everywhere all at once?

Because it worked good for them so far? There are plenty of companies that do the same and we don’t hear about them until something goes wrong.

It seems like a none of the above situation because each of those should have really minimized the chances of something like this happening. But this is pure speculation. Even the most perfect organization engineering culture can still have one thing get through... (Wasn't there some Linux incident a little back though?)

Quality starts with good design, good people, etc. the process parts come much after that. I'd like to think that if you do this "right" then this sort of stuff simply can't happen.

If we have organization/culture/engineering/process issues then we're likely not going to get an in-depth public most-mortem. I'd love to get one just for all of us to learn from it. Let's see. Given the cost/impact having something like the Challenger investigation with some smart uninvolved people would be good.

Was somebody trying to install an exploit or back door and fucked up?

Everything is a conspiracy now eh?

You do remember Solarwinds right? This is an obvious high value target, so it is reasonable to entertain malicious causes.

Given the number of systems infected, if you could push code that rebooted every client into a compromised state you’d still have run of some % of the lot until it was halted. That time window could be invaluable.

Now, imagine if you screw up the code and just boot loop everything.

I’d say business wise it’s better for crowd strike to let people think it’s an own-goal.

The truth may be mundane but a hack is as reasonable a theory as “oops we pushed boot loop code to world+dog”.

> The truth may be mundane but a hack is as reasonable a theory as “oops we pushed boot loop code to world+dog”.

No it's not. There are many signs that point to this being a mistake. There are very few that point to it being a hack. You can't just go "oh it being a hack is one of the options therefore it is also something worth considering".

Look there are two options on the table so it's 50/50. Ipso facto.

I believe the flying spaghetti monster touched the file with His invisible noodly appendage so now it's a three way split.

I didn’t say it was 50/50, but an accurate enumeration of options does include a failed attempt at a hack.

I fail to see why this is so difficult to understand.

Especially because if it was crowdstrike wouldn’t be apologizing and accepting blame.

Why? They are in a very specific business and have more incentive to cover up successful attacks than most other companies.

And while I'm 99% for Hanlon's razor here, I don't see a reason to be sure it wasn't even a completely successful DoS attack.

“Our employee pushed bad code by accident” is VASTLY better for them than “we didn’t secure the infra that pushes updates to millions of machines”.

To be fair, the xd backdoor wasn’t immediately obvious https://www.wired.com/story/xz-backdoor-everything-you-need-...

In a world of complex systems a "confluence of improbable events" is the same thing as "a disaster waiting to happen". Its the swiss cheese model of failure. Y

Every system can only survive so many improbable events. Even in aviation.