Hacker News Reader

Comment by AlbertCory | Hacker News Reader

"Diversity" (but not in the sense of marginalized people)

If more of the critical machines were running different OS's, the damage would be contained.

When we talk about the dangers of "monoculture" it's usually about plants. The same danger applies to computing infrastructure.

View on HN

We're already there. The fact that we didn't see civilization collapse is evidence that there is a ton of infrastructure not running Windows and Crowdstrike.

For a long time after Burroughs was almost ancient history, banks still ran Burroughs machines. They've probably thrown in the sponge by now.

I'm sure IBM mainframes are still running critical stuff, too.

This wasn't nearly as bad as it could have been. What if the crash wasn't just a crash but resulted in data corruption? And what if it took longer to stop the rollout and deploy a fixed version? How long would it have taken to recover from this kind of incident? If affected machines didn't fix themselves after several reboots but needed to be actively reimaged?

On top of that, I am still struggling to understand how the people in charge of running orgs that run highly critical systems were OK with the idea that a 3rd party software provider could push at anytime patches to the software they provide.

Sorry for being harsh with my following statement, but I believe that the companies affected by Crowdstrike share some responsibility on what happened yesterday.

The are OK with "push at anytime patches to the software" because that's a big part of what they are paying for. Rapid response to threats.

>Ping reply from 127.0.0.1

The threat is inside the building!

You're making the mistake of assuming that the people running those companies care about anything other than their job security, and buying in solutions is the best way to have a ready-made scapegoat when things go wrong. The mantra "no-one ever got sacked for buying IBM" still holds, you can just substitute "Oracle", or "Microsoft", or now - apparently - "Crowdstrike".

They share the whole of the responsibility of it. "my antivirus was updating" is not an acceptable excuse for a service to be down.

It's not harsh. The tide went out and it turns out a lot of people were swimming naked.

I think I agree with you. On the other hand, I can also imagine that if autoupdates weren't the case, then 90% of installations would be a terribly outdated and probably vulnerable version. It's hard to imagine a common sense middle ground.

Surprisingly, the mantra "if it works, don't touch it" doesn't really work so great.

One could make the argument that automatically patched software is, in aggregate, more secure/less problematic than chronically under-patched software that requires manual, human attention.

One could, but in the old days when vulnerabilities happened, they didn't hit everyone at once.

And if it hit your system, the vendor's first response would be "are you on the latest update? that's been fixed."

(Unless the latest update IS the problem. In that case, being lazy was a good defense.)

As I understand it, customers do have control, but in this instance CrowdStrike overrode the settings of the customers.

- pushing patches is objectively a good idea, rapid response to threats and all.

- Whats bad is instant global 0->1 rollout, instead of more gradual, blue/green/canary however you call it. With gradual rollout policy this whole thing could have been caught at their first couple guinea pig customers, and not the whole world

You don't understand the word objective. It is beyond arrogant to think that controlling when a customer's day gets ruined is your prerogative. Let them make that decision.

They chose a major vendor and it checks off a compliance requirement.

> If more of the critical machines were running different OS's, the damage would be contained.

Not if they were running the same CrowdStrike.

given it's a kernel module (AFAIK), how could that be if it were different OS's?

Regardless, much code would be shared. Likely including the offending null pointer access of this case.

> Regaress

"regardless" ?

So you don't actually know, is what you're saying.

Thanks. Typo corrected.

Not necessarily. CrowdStrike isn't even the #1 player in this space, but this still happened because of network effects. The number of platforms you'd need for this much safety is impractically high.

I'm not saying you're wrong, but:

"Network effects"? You mean like, "I'd be fine, but I depend on a service from a Windows machine, so I'm still screwed" ?

> The number of platforms you'd need for this much safety is impractically high

I don't see why this becomes an impossible problem. If all the essential services are not provided by a single software infrastructure, then we have the required diversity, right?

> "Network effects"? You mean like...

In the case of airports, losing ATC at just a few major US airports would effectively paralyze the network. Or yes, the case you mentioned where you depend on four SaaS offerings, and odds are one of them will go down.

Like I said: I'm not arguing with you.

It's ironic that the original DARPA justification for packet-switching was, if a nuclear war takes down some nodes, the packets will still get through somehow.

Computers are not people. No need to be afraid to discriminate.

Windows is shit.

Mac is more or less.

Linux is best of all.

You do realize that CrowdStrike also runs on Linux and that there have been a variety of instances of bad CrowdStrike updates breaking Linux machines, right?

https://access.redhat.com/solutions/7068083

Reply on news.ycombinator.com