Hacker News Reader

Comment by nmg | Hacker News Reader

This would answer the question that i've not heard anyone asking:

what incentivized the bad decisions that led to this catastrophic failure?

My understanding is that the culture (as reported by some customers) is quite aggressive and pushy. They are quite vocal when customers don’t turn in automatic updates.

It makes sense in a way - given their fast growth strategy (from nowhere to top 3) and desire to “do things differently” - the iconoclast upstarts that redefine the industry.

Or to summarise - hubris.

To catch 0day quickly, EDR needs to know "how".

The "how" here is AV definition or a way to identify the attack. In CS-speak: content.

Catching 0day quickly results in good reputation that your EDR works well.

If people turn off their AV definition auto-update, they are at-risk. Why use EDR if folks don't want to stop attack quickly?

In theory you're correct. In practice it seems that crowdstrike has crashed systems with their updates much more often than 0day attacks.

How many times?

This is one bsod on windows 10. I saw another kernel panic on specific Linux distro.

What else?

One thing that is funny is that quite a few of their competitors are taking this opportunity to shit on them via Twitter and by marketing themselves as better than CrowdStrike.

Twitter, with all its issues, apparently has a feature to prevent fake news and that feature will show crowd source sentiment to debunk fake news, in this case, Twitter users showed how many times Crowdstrike competitor BSOD windows

Bottom line is this: there is absolutely no good reason for not doing rolling updates. Do a few and make sure they are ok. Keep rolling out in groups. This single approach alone would've meant that this event was of marginal impact to most of the public, as sysadmins would've had the opportunity to halt further updates and work on remediating their first group (typically non-critical servers). Rolling out to everything all at once is just bad practice, period.

Put yourself in the customer shoes: who wants to sign up to be rolled first?

There's best practice and then there's customer

Give those customers a discount.

The solution to customer reluctance to being the guinea pig is not to force every customer to be a guinea pig.

> This is one bsod on windows 10. I saw another kernel panic on specific Linux distro.

The red hat one. But they also did it to debian with a different issue, and I think another distro as well.

> They are quite vocal when customers don’t turn in automatic updates.

I'm sorry but this is the customer's fault.

If I'm using your services you work for me and you don't get to bully me into doing whatever you think needs to be done.

People that chose this solution need to be penalized, but they won't.

Customers don’t always have a choice here. They could be restricted by compliance programs (PCI, et al) and be required under those terms to have auto updates on.

Compliance also has to share some of the blame here, if best practices (local testing) aren’t allowed to be followed in the name of “security”.

This needs to keep being repeated anytime someone wants to blame the company.

Many don’t have a choice, a lot of compliance is doing x to satisfy a checkbox and you don’t have a lot of flexibility in that or you may not be able to things like process credit cards which is kinda unacceptable depending on your company. (Note: I didn’t say all)

CrowdStrike automatic update happens to satisfy some of those checkboxes.

Reply on news.ycombinator.com