Hacker News Reader

Comment by fredoralive | Hacker News Reader

If this comment tree: https://news.ycombinator.com/item?id=41003390 is correct, someone at Crowdstrike looked at their documented update staging process, slammed their beer down, and said: "Fuck it, let's test it in production", and just pushed it to everyone.

View on HN

Which of course begs the question: How were they able to do that? Was there no internal review? What about automated processes?

For an organization it's always the easiest, most convenient answer to blame a single scapegoat, maybe fire them... but if a single bad decision or error from an employee has this kind of impact, there's always a lack of safety nets.

Even if true, the orgs whose machines they are have the responsibility to validate patches.

This is not a patch per se, it was Crowdstrike updating their virus definition or whatever it's called internal database.

Such things are usually enabled by default to auto-update, because otherwise you lose a big part of the interest (if there's any) of running an antivirus.

Surely their should be at least some staging on update files as well, to avoid the "oops, we accidentally blacklisted explorer.exe" type things (or, indeed, this)?

Companies have staging and test process but CS bypassed it and deployed to prod.

If I understand the thread correctly, CS bypassed the organization's staging system

I'm guessing there's a lesson to be learned here.

Reply on news.ycombinator.com