Hacker News Reader

Comment by golemiprague | Hacker News Reader

But how come they didn't catch it in the testing deployments? what was the difference that caused it to happen when they deployed to the outside world. I find it hard to believe that they didn't test it before deployment. I also think companies should all have a testing environment before deploying 3rd party components. I mean, we all install some packages during development that fails or cause some problems but nobody think it is a good idea to do it directly in their production environment before testing, so how is this different?

View on HN

My guess -- there are two separate pipelines one for code changes and one for data files.

Pipeline 1 --

Code updates to their software are treated as material changes that require non-production and canary testing before global roll-out of a new "Version".

Pipeline 2 --

Content / channel updates are handled differently -- via a separate pipeline -- because only new malware signatures and the like are distrubuted via this route. The new files are just data files -- they are supposed to be in a standard format and only read, not "executed".

This pipeline itself must have been tested originally and found tobe working satisfactorily -- but inside the pipeline there is no "test" stagethat verifies the integrity of the data fine so generated, nor - more importantly - checking if this new data file works without errors when deployed to the latest versions of the software in use.

The agent software that reads these daily channel files must have been "thoroughly" tested (as part of pipeline 1) for all conceivable data file sizes and simulated contents before deployment. (any invalid data files should simply be rejected with an error ... "obviously")

But the exact scenario here -- possibly caused by a broken pipeline in the second path (pipeline 2) -- created invalid data files with some quirks. And THAT specific scenario was not imagined or tested in the software version dev-test-deploy pipeine (pipeline 1).

If this is true --

The lesson obviously is that even for "data" only distributions and roll-outs, however standardized and stable their pipelines may be, testing is still an essential part before large scale roll-outs. It will increase cost and add latency sure, but we have to live with it. (similar to how people pay for "security" software in the first place)

Same lesson for enterprise customers as well -- test new distributions on non-production within your IT setup, or have a canary deployment in place before allowing full roll-outs into production fleets.

It was mentioned in one of the HN threads, that the update was pushed overriding the settings customer had [1]. What recourse any customer can have in in such a case ?

1. https://news.ycombinator.com/item?id=41003390

> What recourse any customer can have in in such a case ?

Sue them and use something else.

Ah that was me. We don’t accept “content updates” and they are staged.

We got this update pushed right through.

Nice.

But the problem here is that the code runs in kernel mode. As such any data that it may consume should have been tested with the same care as the code itself which has never been the case in this industry.

> It will increase cost

And of of course that cost would be absolutely insignificant relative to the potential risk...

> I find it hard to believe that they didn't test it before deployment.

I’m not sure why you find that hard to believe - based on the (admittedly fairly limited) evidence we have right now, it’s highly unlikely that this deployment was tested much, if at all. It seems much more likely to me that they were playing fast and loose with definition updates to meet some arbitrary SLAs[1] on zero-day prevention, and it finally caught up with them. Much more likely than somehow every single real-world pc running their software being affected but their test machines somehow all impervious.

[1] When my company was considering getting into endpoint security and network anomaly detection, we were required on multiple occasions by multiple potential clients to provide a 4-hour SLA on a wide number of CVE types and severities. That would mean 24/7 on-call security engineers and a sub-4-hour definition creation and deployment. Yes, that 4 hours was for the deployment being available on 100% of the targets. Good luck writing and deploying a high-quality definition for a zero day in 4 hours, let alone running it through a test pipeline, let alone writing new tests to actually cover it. We very quickly noped out of the space, because that was considered “normal” (at least to the potential clients we were discussing). It wouldn’t shock me if CS was working in roughly the same way here.

This whole f*up was a failure of management and processes at Crowdstrike. "Intern Steve" pushing faulty code to production on a Friday is only a couple of cm of the tip of an enormous iceberg.

I wrote this in another thread already, but the fuck up was both at crowdstrike (they borked a release) but also and more importantly their customers. Shit happens even with the best testing in the world.

You do not deploy anything, ever on your entire production fleet at the same time and you do not buy software that does that. It's madness and we're not talking about small companies with tiny IT departments here.

Shit might happen with the best testing, but with decent testing it would not be this serious.

Apparently CrowdStrike bypassed clients' staging areas with this update.

Source: https://x.com/patrickwardle/status/1814367918425079934

> you do not buy software that does that

Note how the incident disproportionally affected highly regulated industries, where businesses don't have a choice to screw "best practice".

Only highlighting that "best practice" of cybersecurity is, charitably, total bullshit; less charitably, a racket. This is apparent if you look at the costs to the day-to-day ability of employees to do work, but maybe it'll be more apparent now that people got killed because of it.

It’s absolutely a racket.

That’s a tricky one. CrowdStrike is cybersecurity. Wait until the first customer complains that they were hit by WannaCry v2 because CrowdStrike wanted to wait a few days after they updated a canary fleet.

The problem here is that this type of update (a content update) should never be able to cause this however badly it goes. In case the software receives a bad content update, it should fail back to the last known good content update (potentially with a warning fired off to CS, the user, or someone else about the failed update).

In principle, updates that could go wrong and cause this kind of issue should absolutely be deployed slowly, but per my understanding, that’s already the practice for non-content updates at CrowdStrike.

Windows updates are also cybersecurity, but the customer has (had?) a choice to how to roll those out (with Intune nowadays?). The customer should decide when to update, they own the fleet not the vendor!

You do not know if a content update will screw you over and mark all the files of your company as malware. The "It should never happen" situations are the thing you need to prepare for, the reason we talk about security as an onion, the reason we still do staggered production releases with baking times even after tests and QA have passed...

"But it's cybersecurity" is not a justification. I know that security departments and IT departments and companies in general love dropping the "responsibility" part on someone else, but in the end of the day the thing getting screwed over is the company fleet. You should retain control and make sure things work properly, the fact those billion dollar revenue companies are unable to do so is a joke. A terrible one, since IT underpins everything nowadays.

> The customer should decide when to update, they own the fleet not the vendor!

The CS customer has decided to update whenever 24/7 CS says. The alternative is to arrive on Monday morning to an infected fleet.

Sorry, this is untrue. Enterprises have SOCs and oncalls, if there is a high risk they can do at least minimal testing (which would have found this issue as it has a 100% bsod rate) and then fleet rollout. It would have been rolled out by Friday evening in this case without crashing hundred of thousands of servers.

The CS customer has decided to offload the responsibility of its fleet to CS. In my opinion that's bullshit and negligence (it doesn't mean I don't understand why they did it), particularly at the scale of some of the customers :)

> they can do at least minimal testing (which would have found this issue as it has a 100% bsod rate)

Incorrect, I believe, given they did not and could not get advance sight of the offending forced update.

I doubt CrowdStrike had done any testing of the update.

> they can do at least minimal testing (which would have found this issue as it has a 100% bsod rate)

Incorrect, I believe, given they could and did not get advance sight of the offending forced update.

It is a justification, just not necessarily one you agree with.

Companies choose to work with Crowdstrike. One of the reasons they do that is ‘hands-off’ administration-let a trusted partner do it for you. There are absolutely risks of doing it this way. But there are also risks of doing it the other way.

The difference is, if you hand over to Crowdstrike, you’re not on your own if something goes wrong. If you manage it yourself, you’ve only got yourself working on the problem if something goes wrong.

Or worse, something goes wrong and your vendor says “yes, we knew about this issue and released the fix in the patch last Tuesday. Only 5% of your fleet took the patch? Oh. Sounds like your IT guys have got a lot of work on their hands to fix the remaining 95% then!”.

You'd think that the software would sit in a kind of sandbox so that it couldn't nuke the whole device but only itself. It's crazy that this is possible.

The software basically works as a kernel module as far as I understand, I don’t think there’s a good way to separate that from the OS while still allowing it to have the capabilities it needs to have to surveil all other processes.

Something like ebpf.

And even then, you wouldn’t want the system to continue running if the security software crashes. Such a crash might indicate a successful security breach.

> You do not deploy anything, ever on your entire production fleet at the same time and you do not buy software that does that

I am sympathetic to that, but its only possible if both policy and staffing allow.

for policy, there are lots of places that demand CVEs be patched within x hours depending on severity. A lot of times, that policy comes from the payment integration systems provider/third party.

However you are also dependent on programs you install not autoupdating. Now, most have an option to flip that off, but its not always 100% effective.

> I am sympathetic to that, but its only possible if both policy and staffing allow.

We are not talking about small companies here. We're talking about massive billion revenue enterprises with enormous IT teams and in some cases multiple NOCs and SOCs and probably thousands consultants all around at minimum.

I find it hard to be sympathetic to this complete disregard of ownership just to ship responsibility somewhere else (because this is the need at the of the day let's not joke around). I can understand it, sure, and I can believe - to a point - someone did a risk calculation (possibility of crowdstrike upgrade killing all systems vs hack if we don't patch a CVE in <4h), but it's still madness from a reliability standpoint.

> for policy, there are lots of places that demand CVEs be patched within x hours depending on severity.

I'm pretty sure leadership when they need to choose between production being down for an unspecified amount of time and taking the risk of delaying (of hours in this case) the patching will choose the delay. Partners and payment integration providers can be reasoned with, contracts are not code. A BSOD you cannot talk away.

Sure, leadership is also now saying "but we were doing the same thing as everyone else, the consultants told us to and how could have we have known this random software with root on every machine we own could kill us?!" to cover their asses. The problem is solved already, since it impacted everyone, and they're not the ones spending their weekend hammering systems back to life.

> However you are also dependent on programs you install not autoupdating. Now, most have an option to flip that off, but its not always 100% effective.

You choose what to install on your systems, and you have the option to refuse to engage with companies that don't provide such options. If you don't, you accept the risk.

Disagree with the part where you put onus on customer. As has been mentioned in other HN thread [1], this update was pushed ignoring whatever the settings customer had configured. The original mistake of the customer, if any, was they didn't read this in fine print of the contract (if this point about updates was explicitly mentioned in the contract). 1. https://news.ycombinator.com/item?id=41003390

> You do not deploy anything, ever on your entire production fleet at the same time

And if an attacker does??

Oh absolutely. There’s many levels of failure here. A few that I see as being likely:

- Lack of testing of a deployment - Lack of required procedures to validate a deployment - Engineering management prioritizing release pace over stability/testing - Management prioritizing tech debt/pentests/etc far too low - Sales/etc promising fast turnarounds that can’t be feasibly met while following proper standards - Lack of top-down company culture of security and stability first, which should be a must for any security company

This outage wasn’t caused only by “the intern pushing release.” It was caused by a poor company culture (read: incorrect direction from the top) resulting in a lack of testing of the program code, lack of testing environment for deployments, lack of formal deployment process, and someone messing up a definition file that was caught by 0 other employees or automated systems.

I can't speak to its veracity but there's a screenshot making its way around in which Crowdstrike discouraged sites from testing due to the urgency of the update.

I don’t work with CS products atm, but my experience with a big CS deployment was exactly like this. They were openly quite hostile to any suggestion of testing their products, we were frequently rebuked for running our prod censors on version n-1. I talked about it a bit in this comment.

https://news.ycombinator.com/item?id=%2041002864

Very much not surprised to see this now.

It’s kind of hard to pitch “zero-day prevention” if you suggest people roll out definitions slowly, over the course of days/weeks. Thus making it a lot harder to charge to the moon for your service.

Now, if these sorts of things were battle tested before release, and had a (ideally decade+-long) history of stability with well-documented processes to ensure that stability, you can more easily make the argument that it’s worth it. None of those things are close to true though (and more than likely will never be for any AV/endpoint solution), so it is very hard to justify this sort of configuration.

While true agent should roll back to previous content version if it keeps crashing

Detecting system crashes would be hard. You could try logging and comparing timestamps on agent startups and see if the difference is 5 minutes or less. Buggy kernel drivers crash Windows hard and fast.

loading content is pretty specific step so your solution is more or less valid

> Detecting system crashes would be hard.

Store something like an `attemptingUpdate` flag before updating, and remove it if the update was successful. Upon system startup, if the flag is present, revert to the previous config and mark the new config bad.

One possible explanation could be automated testing deployments for definitions updates that don't run the current version of the definition consumer, and the old one they do run is unaffected.

I’ve seen places where failed releases are just “part of normal engineering.” Because no one is perfect, they say.

I really dislike this mentality. Don't even get me started on celebrating when your rocket blows up

If it is a standard production rocket, I agree. If it is a first of kind or even third of kind launch, celebrating the lessons learned from a failure is a healthy attitude. This production software is not the same thing at all.

spaceX celebrating when their rocket blows up after a certain milestone it's like us devs celebrating when our branch with that new big feature only fails a few tests. Did it pass no? Are you satisfied as first try? Probably

Even on hn, comments advocating engineering excellence or just quality in general are frequently looked down on, which probably also tells you a lot about the wider world.

This is why we can’t have nice things, but maybe we just don’t want them anyway? “Mistakes will be made” is way less true if you actually put the effort in to prevent them, but I am beginning to think this has become code for quiet-quitters to telegraph a “I want to get paid for no effort and sympathize with others who feel the same” sentiment and appear compassionate and grimly realistic all at the same time.

yes, billion dollar companies are going to make mistakes, but almost always because of cost cutting, willful ignorance, or negligence. If average people are apologizing for them and excusing that, there has to be some reason that it’s good for them.

Personally while I value excellence, I reduce the frequency of errors through process and procedure because I’m lazy.

I don’t mind meetings but being in a 4 hour emergency meeting because some due diligence wasn’t done is a waste of my time.

Life is easier when you do good work.

I find it hard to believe they didn't do any testing. I wonder if they tested the virus signatures against the engine, but didn't check the final release artefact (the .sys file) and the bug was somehow introduced in the packaging step.

This would have been poor, but to have released it with no testing would have been the most staggering negligence.

That’s what a lot of us are wondering. There’s a lot of outside thinking of the box right now about this in certain circles.

There’s no point in leaving vague allusions. Can you expand on this?

security industry's favorite language is nothingspeak

for all we know, the deployment was the test

As the old saying goes, everyone has a test environment, and some also have a separate production one.

Reply on news.ycombinator.com