The CrowdStrike file that broke everything was full of null characters?
twitter.com
twitter.com
I’m not even particularly stringent when it comes to automated test across-the-board but for this level of criticality of system, you need exceptionally good state management
To the point where you should not roll to production without an integration test on every environment that you claim to support
Like it’s insane to me that this size and criticality of a company doesn’t have a staging or even a development test server that tests all of the possible target images that they claim to support.
Who is running stuff over there - total incompetence
We don't have the answers, but I'm not in a rush to assume that they don't test anything they put out at all on Windows.
They would be a gigantic target for malware. Imagine pwning a CDN to pwn millions of client computers. The CDN being malicious would be a major threat.
Antiviruses are security cosplay that works by using a combination of bug-riddled custom kernel drivers and unsandboxed C++ parsers running with the highest level of privileges to tamper with every bit of data it can get its hands on. They violate every security common sense. They also won't even hesitate to disable or delay rollouts of actual security mechanisms built into browsers and OSes if it gets in the way.
The software industry needs to call out this scam and put them out of business sooner than later. This has been the case for at least a decade or two and it's sad that nothing has changed.
https://ia801200.us.archive.org/1/items/SyScanArchiveInfocon... https://robert.ocallahan.org/2017/01/disable-your-antivirus-...
Antivirus software is almost universally malware. Enterprise endpoint "protection" software like CrowdStrike is worse, it's an aggressive malware and a backdoor controlled by a third party, whose main selling points are compliance and surveillance. Installing it is a lot like outsourcing your secret police to a consulting company. No surprise, everything looks pretty early on, but two weeks in, smart consultants rotate out to bring in new customers, and bad ones rotate in to run the show.
Yeah, that's definitely a good tradeoff against script kiddies metasploiting your ordinary finance employees. Wonder if it'll look as good when loss of life caused by CrowdStrike this weekend gets tallied up.
The ends don’t justify the means.
Yes, occasionally they do. This is not an either-or situation.
While they do catch and stop attacks, it is also true that crowdstrike and its ilk are root-level backdoors into the system that bypass all protections and thus will cause problems sometimes.
Options include:
1. protected the systems always work even if things are messed up
2. protected systems are always protected even when things are messed up
The two failure modes are exclusive; ideally you let the end user decide what to do if the protection mechanism is itself unstable.
One could suggest "the system must always work" but that's ignoring that sometimes things don't go to plan.
None of the systems in boot loops were p0wned by known exploits while they were boot looping. As far as we know anyhow.
(edited to add the obvious default of "just make a working system" which is of course both a given and not going to happen)
This isn’t a matter of “either your system is protected all the time, even if that means it’s down, or your system will remain up but might be unprotected.” It’s “your system is vulnerable to kernel-level exploits because of your AV software’s inability to validate definition files.”
The failure mode here should absolutely not be to soft-brick the machine. You can have either of your choices configurable by the sysadmin; definition file fails to validate? No problem, the endpoint has its network access blocked until the problem can be resolved. Or, it can revert to a known-good definition, if that’s within the organization’s risk tolerance.
But that would require competent engineering, which clearly was not going on here.
How about a different analogy: First do no harm.
Processes need to match the potential risk.
If your company is doing some inconsequential social app or whatever, then sure, go ahead and move fast and break things if that's how you roll.
If you are a company, let's call them Crowdstrike, that has access to push root privileged code to a significant percentage of all machine on the internet, the minimum quality bar is vastly higher.
For this type of code, I would expect a comprehensive test suite that covers everything and a fleet of QA machines representing every possible combination of supported hardware and software (yes, possibly thousands of machines). A build has to pass that and then get rolled into dogfooding usage internally for a while. And then very slowly gets pushed to customers, with monitoring that nothing seems to be regressing.
Anything short of that is highly irresponsible given the access and risk the Crowdstrike code represents.
That doesn't work in the business they're in. They need to roll out definition updates quickly. Their clients won't be happy if they get compromised while CrowdStrike was still doing the dogfooding or phased rollout of the update that would've prevented it.
Well clearly we have incontrovertible evidence now (if it was needed) that YOLO-pushing insufficiently tested updates to everyone at once does not work either.
This is being called in many places (righfully) the largest IT outage in history. How many billions will be the cost? How many people died?
So yes, clearly not the correct way to operate.
Its negligence.
This is a public company after all. In this market, you don’t become a “Top-Tier Cybersecurity Company At A Premium Valuation” with amazing engineering practices.
Priority is sales, increasing ARR, and shareholders.
This isn’t hindsight. It’s “don’t blow up 101” level stuff they messed up.
It’s not that this got past their basic checks, they don’t appear to have had them.
So let’s ask a different question:
The file parser in their kernel extension clearly never expected to run into an invalid file, and had no protections to prevent it from doing the wrong thing in the kernel.
How much you want to bet that module could be trivially used to do a kernel exploit early in boot if you managed to feed it your “update” file?
I bet there’s a good pile of 0-days waiting to be found.
And this is security software.
This is “we didn’t know we were buying rat poison to put in the bagels” level dumb.
Not “hindsight is 20/20”.
Also, the tech industry is extremely competitive. Leaders frequently become laggards or go out of business. Here are some companies who failed or shrank because their products could not complete: IBM, Digital Equipment, Sun, Borland, Yahoo, Control Data, Lotus (later IBM), Evernote, etc. Note all of these companies were at some point at the top of their industry. They aren't anymore.
Eventually
By then the principles are all very rich, and no longer care.
Do you think Bill Gates sleeps well?
IMO point is not where are these past top companies now but where are top people in those companies now. I believe they end up being in very comfortable situation no matter which place.
Exceptions of course would be criminal prosecution, financial frauds etc.
Eventually is a long time.
Unfortunately for all of us ("us" being not just software engineers, but everyone impacted by this and similar lack of proper engineering outcomes) it is a proven path to wealth and success to ignore engineering a good product. Build something adequate on the surface and sell it like crazy.
Yeah, eventually enough disasters might kill the company. Countless billions of dollars will have been made and everyone responsible just moves on to the next one. Rinse & repeat.
It's very possible the signature validation and verification happens after the bug was triggered.
(...which doesn't rule out the possibility that CS was doing it.)
Pretty sure "competence" wasn't part of the job description of the ClownStrike developers, at least for those pieces. :( :( :(
Are those drivers signed? Who can sign them? Only Microsoft?
If it's true the file contained nothing but zeros that seems to be also kernel vulnerability. Even if signing were not mandatory, shouldn't the kernel check for some structure, symbol tables or the the like before proceeding?
Choice #1 Diable security software and continue. Choice #2 Stop. BSOD message contact you administrator
There may be nothing wrong with the drivers.
(This is exactly how CPU microcode updates work — the CPU “takes receipt” of the new microcode package, and integrity-verifies it internally, before starting to do anything involving updating.)
Fantastic solution! You just gave the attackers a way to stop all security updates to the system.
For most systems, a sensible algorithm would be "keep running the last known good definition, until we get the next known good definition"
In other words: ignore bad updates but keep checking for valid ones. That doesn't mean you've permanently lost the ability to update.
Of course, for some systems, different behavior might make more sense.
Brillant! [sic]
And yes Windows drivers are signed. If it had been a driver it would just have failed to load. Nowadays they must be signed by Microsoft, see https://learn.microsoft.com/en-us/windows-hardware/drivers/d...
The kernel driver was signed. The file it loaded as input with garbage data had seemingly no verification on it at all, and it crashed the driver and therefore the kernel.
https://learn.microsoft.com/en-us/windows-hardware/drivers/d...
I’m not surprised that there is no test pipeline for new data files. Those aren’t even really “build artifacts.” The software assumes they’re just data.
But I am surprised that the kernel module was deployed with a bug that crashed on a data file with all nulls.
(In fact, it’s so surprising, that I wonder if there is a known failing test in the codebase that somebody marked “skip” and then someone else decided to prove a point…)
Btw: is that bug in the kernel module even fixed? Or did they just delete the data file filled with nulls?
Is that a real question? They definitely didn't do anything more than delete the file, perhaps just rename it.
1. Start Windows in Safe Mode or the Windows Recovery Environment (Windows 11 option).
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
3. Locate the file matching C-00000291*.sys and delete it.
4. Restart your device normally.I.e. only one link in the chain wasn't tested.
Sorry, but that will not do.
> We don't have the answers, but I'm not in a rush to assume that they don't test anything they put out at all on Windows.
The parent post did not suggest they don't test anything. It suggested they did not test the whole chain.
> it’s insane to me that this size and criticality of a company doesn’t have a staging or even a development test server that tests all of the possible target images that they claim to support
I know nothing about Crowdstrike, but I can guarantee that "they need to test target images that they claim to support" isn't what went wrong here. The implication that they don't test against Windows is so incredulous, it's hard to take the poster of that comment seriously.
The fact is mistakes happen (even huge ones), and the best thing to do is learn from the mistakes. The other thing people seem to forget is they are probably doing a lot of the same things which got CrowdStrike into trouble.
If I had to guess, one problem may be that CrowdStrike's Windows code did not validate the data it received from the update process. Unfortunately, this is very common. The lesson is to validate any data received from the network, from an update process, received as user input, etc. If the data is not valid, reject it.
Note I bet at least 50% of the software engineers commenting in this thread do not regularly validate untrusted data.
And given Crowdstrike are, and data validation neglect is so common, why have they not already learned this lesson?
It was ages ago, but from what I remember, the first day rollout did occasionally catch issues. It only affected a small number of users and the risk was within the tolerance window.
We also tested locally before the first rollout.
Not thay it exuces anything, just that this probably wasn't treated as an update at all.
It's just too early to tell what happened here.
The likelihood is that it _was_ negligence. But we need a proper post-mortem to be able to determine one way or another.
It doesn't matter how much you test if something down the line zeroes out your stuff.
Our release process is more like: build and package, sign package, run CI tests on signed package, run manual tests on signed package, release signed package. The deployment process should check those signatures. A test process should by design be able to detect any copy errors between test and release in a safe way.
But at this stage, all we are doing is speculating.
if(dataFileIsNotValid) { throw FatalKernelException("All your base are compromised"); }
EDIT+ Explanation:
With hindsight not booting may be exactly the right thing to do since a bad datafile would indicate a compromised distribution/ network.
The machines should not fully boot until file with valid signature is downloaded.*
Who is saying they don't have that? Who is saying it didn't pass all of that?
You're making tons of assumptions here.
Presumably crowdstrike employs some smart engineers. I think it’s reasonable to assume that those engineers know what CI/CD is, they understand its utility, and they’ve used it in the past, hopefully even at Crowdstrike. Assuming that this is the case, then how does a bug like this make it into production? Why aren’t they doing the things that would have prevented this? If they cut corners, why? It’s not useful or productive to throw around accusations or demands for specific improvements without answering questions like these.
Call me old-fashioned and boring, but I'd like to have some basic facts about the situation first. After this I decide who does and doesn't deserve a bollocking.
Thay info includes that the faulty file consisted entirely of zeros.
Even that is not certain. Some people are reporting that this isn't the case and that the all-zeroed file may be a "quick hack" to send out a no-op.
So no, we have very little info.
Bottom line this was avoidable and negligent
For the record I owned global infrastructure as CTO for the USAF Air Operations weapons system - one of the largest multi-classification networked IT systems ever created for the DoD - even moreso during a multi-region refactor as a HQE hire into the AF
So I don’t have any patience for millionaires not putting the work in when it’s critical infrastructure
People need to do better and we need accountability for people making bad decisions for money saving
Lots of very smart people have been hard at work to prevent airplanes from crashing for many decades now, and planes still crash for all sorts of reasons, usually considered "avoidable" in hindsight.
Nothing is "bulletproof"; this is a meaningless buzzword with no content. The world is too complex for this.
There is no such thing.
What are your thoughts on MSFTs role in this?
They’ve been iterating Windows since 1985 - doesn’t it seem reasonable that their kernel should be able to survive a bad 3rd party driver?
2. The problem is the ubiquity of windows so it’s embedded in the infrastructure
We’ve put too many computers in charge of too much stuff for the level of combined capabilities of the computer and the human operator interface
I'm not sure: is having test servers that it passed any better than none at all?
One uncaught downstream failure doesn't invalidate the effort into all the previously caught failures.
In this case it gives me vibes of something going wrong after the CI pipeline, during the rollout. Maybe they needed advice a bit more specific than "just use a staging environment bro", like "use checksums to verify a release was correctly applied before cutting over to the new definitions" and "do staged rollouts, and ideally release to some internal canary servers first".
I honestly though HN was slightly higher quality than most of the comments here. I am proven wrong.
HN reminds me of nothing so much as Slashdot in the early 2000's, for both good and ill. Fewer stupid memes about Beowulf Clusters and Natalie Portman tho.
But naw, must be every random commentator on HN knows how to run the company better.
Wonder if the higher-ups who mandated this software to be installed in their hospitals were informed about that fact.
For all we know there is a test case that failed and they decided to push the module anyway (“it’s not like anyone is gonna upload a file of all nulls”).
Btw: where are these files sourced from? Could a malicious Crowdstrike customer trick the system into generating this data file, by e.g. reporting it saw malware with these (null) signatures?
You sound like the guy that a few years ago tried to argue (the company in question) tested os code that didn't include any drivers for their gear's local storage. Its obvious it wasn't to anyone competent.
Linux and Unix ecosystems in general work by people thoroughly testing and taking responsibility for their work.
Windows ecosystems work by blame passing. Blame Ron, the IT guy. Blame Windows Update. Blame Microsoft. That's how stuff works.
It has always worked this way.
But also, all the good devs got offered 3X the salary at Google, Meta, and Apple. Have you ever applied for a job at CrowdStrike? No? That's why they suck.
* A disproportionately large number of Windows IT guys are named Ron, in my experience.
It's possible that they run tests on new commits, but not when some other, external, non-git system pushes out new data.
Team A thinks that "obviously the driver developers are going to write it defensively and protect it against malformed data", team B thinks "obviously all this data comes from us, so we never have to worry about it being malformed"
I don't have any non-public info about what actually happened, but something along these lines seems to be the most likely hypothesis to me.
Edit: Now what would have helped here is a "staged rollout" process with some telemetry. Push the update to 0.01% of your users and solicit acknowledgments after 15 minutes. If the vast majority of systems are still alive and haven't been restarted, keep increasing the threshold. If, at any point, too many of the updated systems stop responding or indicate a failure, immediately stop the rollout, page your on-call engineers and give them a one-click process to completely roll the update back, even for already-updated clients.
This is exactly the kind of issue that non-invasive, completely anonymous, opt-out telemetry would have solved.
Or even at the very least the most popular OS that they support. I'm genuinely imagining right now that for this component, the entirety of the company does not have a single Windows machine they run tests on.
IMO, I think a root cause issue is that the "hacker types" who are most likely to want to start security software companies are also the least likely to want to implement the "boring" pieces of a process-oriented culture. For example, I can't speak so much for CrowdStrike, but it came out the SolarWinds had an egregiously bad security culture at their company. When the root cause comes out about this issue dollars-to-donuts it was just a fast and loose deployment process.
But I see it as the XKCD 2347 problem.
To emphasize, I'm really just thinking about the bad practices that were reported after the SolarWinds hack (password of "solarwinds123" and a bunch of other insider reports), so I can't say that totally applies to CrowdStrike, but in general I don't feel like these companies that can have such a catastrophic impact take appropriate care of their responsibilities.
When I heard CrowdStrike was here, it just kinda made sense
As they got larger, they added a Sunnyvale office, and later moved the official headquarters to Austin, TX.
They've also been expanding their engineering operations overseas which likely includes offshoring in the last few years.
Most other software doesn't need that low level of access, and even if it crashes, it doesn't take the whole system with it, and a quick, automated upgrade process is possible.
A good developer with access to the kernel can create "better" security software which does less context switching and has less performance impact. But a bad (incompetent or malicious) developer can do a lot more harm with direct access to the kernel.
We see the exact same reasoning with restricting execution of JIT-compiled code in iOS.
I disagree, security companies suffer from "too big to fail" syndrome where the money comes easy because they have customers who want to check a box. Security is NOT a product you pay for, it's a culture that takes active effort and hard work to embed from day one. There's no product on the market that can provide security, only products to point a finger at when things go wrong.
- CrowdStrike
- SolarWinds
- Kaspersky
- https://en.wikipedia.org/wiki/John_McAfee
I'm sure I'm forgetting a few. Maybe it's the same self-selection scheme that afflicts the legal cannabis industry?
No corporate org is gonna bat an eye at Riot's anti-cheat practices, because they aren't installing LoL on their line of business machines anyway.
But if their business is incompatible with strict software whitelisting, their employees might...
The incentives and priorities are very different for people who want to play fair games than they are for people who want to maximize their own freedom.
A modern game need only telemetry that captures what a spectating admin picks up, rather than active surveillance.
Hackers are only a problem when servers are left unmoderated and players can’t vote kick.
If you allow users to kick opponents for cheating, they will have an incentive to kick legitimate players who are playing well.
You cannot have a global competitive ladder if the ladder is split into a ton of different admin-run servers. overwatch had a version of this problem where people would queue into the Australia region during low traffic times to boost their rank.
Suddenly vote kicking had to go, because it was abused. Not in the tournaments themselves, but in open ranked play which serves as qualifiers. An active game can rack up thousands of hours of gameplay per day, far beyond the ability of competent admins to validate. Especially because cheating is often subtle. An expert can spend more than real time looking for subtle patterns that automated tools haven't been built to detect.
Games aren't between you and your 25 buddies for bragging rights anymore. They're between you and 50k other active players for cash prizes. The world has changed. Anti-cheat technology followed that change.
But I strongly disagree about the use of invasive client-side anti-cheat. Server-side anti-cheat can reduce the number of cheaters to an acceptably low level.
See for example how lichess detects and aids in detection of cheaters: https://github.com/clarkerubber/irwin
And chess is a game where I feel like it would be relatively hard to detect cheating. An algorithm looking at games with actors moving in 3D space and responding to relative positions and actions of multiple other actors should have a great many more ways to detect cheating over the course of many games.
Anti-cheat beyond a very basic level is nothing to these companies except a funnel optimization to extract the maximum lifetime value out of the player base. Only the most successful games will ever have the money or reach the technical capability to support this. Nobody making these decisions is doing it for player welfare.
IE the server sends locations and details about all players to your client, even if you are in the spawn room and can't see anyone else and your client has to hide those details from you. It is then trivial to just pull those details out of memory.
The solution forever has been to just not send clients information they shouldn't have. My copy of CS:GO should not know about a terrorist on the other side of the map. The code to evaluate that literally already exists, since the client will answer that question when it goes to render visuals and sound. They just choose to not do that testing server side.
Aimbotting however is probably impossible to stop. Your client has to know where the model for an enemy is to render it, so you know where the hitbox roughly should be, and most games send your client the hitbox info directly so it can do predict whether you hit them. I don't think you can do it behaviorally either.
I don’t believe server side anti cheat can truly be effective against some cheats. But also Vanguard is trash and makes my computer bluescreen. I’ve stopped playing league entirely because of it.
E.g. For CS:GO, the volume of footsteps and gunshots vary by distance so you could use them to triangulate an enemy's position.
Some information is required to cover the network and server delays.
The client predicts what things should look like and then corrects to what they actually are if there is a discrepancy with the server. You cannot get around this short of going back to in-person LAN games.
Just some raycasts through the geometry should be sufficient, which the server is already doing (albeit on likely-simplified collision meshes) constantly.
If you really do have a scenario where occlusion noticeably depends on more of the rendering pipeline (a window that switches between opaque and transparent based on a GPU shader?) you could just treat it as always transparent for occlusion checking and accept the tiny loss that wallhackers will be able to see through it, or add code to simulate that server-side and change the occlusion geometry accordingly.
Here is one for CS:GO, server occlusion geometry must be made separately as auto generating from map is another hard problem.
Not "just some raycasts" otherwise everyone would be doing it:
Only needing raycasts through the geometry was in response to the idea that the server would need to "render the 3d world from each players perspective to do these tests". I don't intend to claim that it's as a whole an easy engineering problem, but rather that the tests don't need to involve much/any of the rendering pipeline.
Of course, no one handles hacking like this because its cheaper to just ship fast and early and never maintain your servers. Not even valve cares about their games and they are the most benevolent company in the industry.
Even if the FPS had no sound, an aimbot-free hacked client that knows the positions of only the players that you can see would still provide a significant benefit because a high fidelity radar and pointers can be placed on the screen. No one can sneak up on you. And no you can't base it on which way the player is looking because you can turn around faster than network latency.
Can you limit the impact of hacked clients? Sure, people might not be able to do egregious stuff like X-ray your position from across the map. Locally, though, game clients need a large amount of information to provide a good experience.
Ultimately even games with kernel anticheat have cheating issues; at some point you have to accept that you cannot stop 100.0% of cheaters. The solution to someone making an aimbot using a physically separate device (reading monitor output, giving mouse input) cannot be to require keys to the player's house.
I think the problem with this is sounds like footsteps or weapons being fired that need to be positional.
Which makes me wonder if you could get away with mixing these sounds server-side and then streaming them to the client to avoid sending positions. Probably infeasible in practice due to latency and game server performance, but fun to think about.
It may still be possible to get round this by using your own machine owner key or using PreLoader/shim [1] to sign a hacked Windows kernel
[1] https://wiki.archlinux.org/title/Unified_Extensible_Firmware...
It does not result in any server-side-detectable difference in behavior other than the hacker seemingly being more aware of their surroundings than they should, which can be hard to determine for sure. Depending on how the hack is done, it may not be detectable by the client either, eg by intercepting the GPU driver calls to render the outlines or switch the wall textures.
in a market where companies compete by impressing nontechnical upper management with presentations, it should be no surprise that technically competent companies have no advantage over incompetent ones
i recently read through the craig wright decision https://www.judiciary.uk/judgments/copa-v-wright/ (the guy who fraudulently claimed to be satoshi nakamoto) and he lacked even the most basic technical competence in the fields where he was supposedly a world-class specialist (decompiling malware to c); he didn't know what 'unsigned' meant when questioned on the witness stand. he'd been doing infosec work for big companies going back to the 90s. he'd apparently been tricking people with technobabble and rigged demos and forged documents for his entire career
george kurtz, ceo and founder of crowdstrike, was the cto of mcafee when they did the exact same thing 14 years ago: https://old.reddit.com/r/sysadmin/comments/1e78l0g/can_crowd... https://en.wikipedia.org/wiki/George_Kurtz
crowdstrike itself caused the same problem on debian stable three months ago: https://old.reddit.com/r/debian/comments/1c8db7l/linuximage6...
it's horrifying that pci compliance regulations have injected crowdstrike (and antivirus) into virtually every aspect of today's it infrastructure
But that's government regulation for you.
since, instead, every business that uses credit cards at all is required to use the same set of counterproductive 'security' policies that make them more vulnerable than before, there is no way for market reality to ground these 'security' policies in real-world security. that's exactly the same problem that happens with government regulation
Anyway, there isn't total solidarity within the credit card cartel.
In a world where it's impossible to support all six card types directly with one codebase, one of the key dimensions on which payment processors would compete for mindshare, would be abstracting away those difficulties in supporting all six card types, enabling the business that uses them to accept more cards, more easily, than if the business went with some rival payment-processor. This would be a headline feature, however hard it is for them to accomplish internally.
The only difference from today, would be that now, due to those potentially-conflicting requirements, the payment processors would have to build six entirely-distinct payment backends that don't share a data model. (Basically treating each credit card platform's cards as different "payment rails", as distinctive from one-another as credit card payment rails today are from debit card payment rails.)
And then, of course, the payment processors themselves would come up with guidelines for handling those six different data warehouses and associated infrastructure, separate/downstream from the guidelines enforced upon each data warehouse by the associated credit platform... and those payment-processor-produced guidelines, might look again quite a lot like PCI-DSS :P
You misspelled “private sector”. Use of endpoint monitoring software is coming out of private auditing companies driven by things like PCI or insurers’ requirements – almost nobody wants to pay for highly-skilled security people so they’re outsourcing it to the big auditing companies and checklists so that if they get sued they can say they were following industry practices and the audit firms okayed it.
I suppose one way to "stop breaches" is to shut down every host entirely.
In the military world, there is a concept of an "Alpha Strike" which generally relates to a fast-enough and strong-enough first-strike that is sufficient to disable the adversary's ability to respond or fight back (e.g. taking down an entire fleet at once). Perhaps people that have been burned by this event will start calling it a Crowdstrike.
Of course, we also need to first start punishing individuals for intentionally causing harm through their decisions even if the harm was caused indirectly through other people. Power allows people to distance themselves from the act. Distance should not affect the punishment.
Yes
But competence is marketed
The trade names like "Crowdstrike" and "Microsoft "
Legal teams are way ahead of you here, by design. They’ve learned to factor out responsibility in such a diffuse manner the “indirectly” term loses nearly all its power the closer one gets to executive ranks and is a fig leaf when you reach C-levels and BoD levels. We couldn’t feasibly operate behemoth corporations otherwise in litigious environments.
I personally suspect we cannot enshrine earnestness and character as metrics and KPI’s, even indirectly. From personal experience selling into many client organizations and being able to watch them make some sausage, this is where leadership over managing becomes a key distinction, and organizational culture over a long period measured in generations of decisions and actions implemented by a group with mean workforce tenure measured in decades becomes one of the few reliable signals. Would love to hear others’ experiences and observations.
> intentionally causing harm
Maybe we need to criminalize groundless accusations too.
Failures like this aren't an individual programmer's responsibility. I can guarantee you programmers have been asking to implement staggered releases and management shut them down. In fact another poster in this thread showed that even customers asked for that and were rejected with 50 pages of corporate BS which basically amounted to "we don't wanna".
Same goes for other safety mechanisms like testing.
> Maybe we need to criminalize groundless accusations too.
I never accused them of intentionally causing harm. Please read my comment more carefully. I said what i said because in the current environment, even doing this intentionally would not lead to sufficient punishment to top management.
I find it amusing that the people commenting on that link are offended this called a "Microsoft " outage, when it is "Crowdstrike's fault".
This is just as much a Microsoft failure.
This is even more, another industry failure
How many times does this have to happen before we get some industry reform that lets us do our jobs and build the secure reliable systems we have spent seven decades researching?
1988 all over again again again
Crowdstrike runs on MacOS and Linux workstations too. And it's just as dangerous there; the big thread has stories of Crowdstrike breaking Linux systems in the past months.
Crowdstrike isn't needed by/for Windows, it's mandated by corporate and government bureaucracies, where it serves as a tool of employee control and a compliance checkbox to check.
That's why it makes no sense to blame Microsoft. If the world run on Linux, ceteris paribus, Crowdstrike would be on those machines too, and would fuck them up just as bad globally.
CrowdStrike _is_ running on a massive number of Linux servers/endpoints, with the same kind of auto-update policies.
This is what chills me to the bone, there's loads of these installations worldwide on heterogeneous OSs but with very little oversight of the code. Companies have basically rolled over and stated, 'OK, we trust you'
I'm not usually a fan of strident calls to open source everything, but the source code at least for the channel file parser on all OSs should now be made public so that we can have an oversight of what so many have placed their trust in.
it's not microsoft's fault at all; crowdstrike caused the same problem on debian systems three months ago. the only way it could be microsoft's fault is if letting users install kernel drivers is microsoft's fault
Just the same week Kaspersky gets kicked from the us market...
Pretty ironic Craig also suggests to put digital certificates on the blockchain instead of them issued by a computer or a cluster of computers, in one place, which makes them a target. A peer to peer network is much more resilient to attacks.
Was the problem, or a part of the problem, that the root certificate of all computers was replaced by CrowdStrike? I didn't follow that part closely. If it was, then certificates registered on the blockchain might be a solution.
I'm totally not an expert in cryptography, but that seems plausible.
As a side note, it is only when i heard Craig's Wright explanation of the blockchain that the bitcoin whitepaper started making a lot of sense. He may not be the greatest of coders, but are mathematicians useless in security?
If it is the certificate, that could be solved in the future by putting the certificates on the blockchain. If it is just incompetent management, then i don't see a great deal of technical solutions.
Craig Wright, claims to be a mathematician not a stellar coder.
Finding a file full of zeroes on a broken computer does not mean it was shipped as all zeroes!
In testing, the dev may have worked from their primary to deploy the update to a series of secondary drives, then sequential performed a test boot from each secondary drive configured for each supported OS version. A shortcut/quick way to test that would've bypassed how their product updates in customer environments, also bypassing checks their software may have performed (in this case, overwriting their own file's contents).
https://www.crowdstrike.com/blog/tech-analysis-channel-file-...
Apparently, Crowdstrike's test setup was fine for this configuration data itself, but they didn't catch it before it was sent out in production, as they were testing the wrong thing. Hopefully they own up to this, and explain what they're going to do to prevent another global-impact process failure, in whatever post-mortem writeup they may release.
Why is this stupid? It’s pretty useful to see a graph of coolant temp vs ambient temp vs motor speed vs roll/pitch.
I must be especially stupid I suppose. Nuts.
He was not my favourite colleague.
I work 9 to 5 Monday to Friday
And post-processing can mean many things. Could be something relatively simple such as "testing passed, so lets mark the file with a version number and release it".
I'd argue you shouldn't even do that. When I've been building CI systems pushing stuff to the customer in the past we've always been automatically versioning everything, and the version that's in the artifact you've found to be good is just the one you're going to be releasing.
Seriously, I don't even know how to do the math on the amount of damage this caused (not including the TIME wasted for businesses as well as ordinary people, for instance those taking flights)
There has to be consequences for this kind of negligence
>Hopefully they own up to this, and explain what they're going to do to prevent another global-impact process failure
They probably needn't bother, every competent sysadmin from Greenland to New Zealand is probably disabling the autoupdate feature right now, firewalling it off and hatching a plan to get the product off their server estate ASAP.
Marketing budgets for competing product are going to get a bump this quarter probably.
https://cyberplace.social/@GossiTheDog/112812454405913406
(scroll down a little)
Most likely is not loaded as a driver binary, but instead is some data file used by the CrowdStrike driver.
Someone may have hoped that overwriting the bad file with an all-0 file of the correct size would make the update benign.
Or following the "QA was bypassed because there was a critical vulnerability" hypothesis, stopping distribution of the real patch may be an attempt to reduce access to the real data and slow reverse-engineering of the vulnerability.
What actually happened was two-folded:
Step 1: CSAgent.sys is a certified durgasoft kernel driver that parses clownstrike virus definition files
Step 2: Clownstrike never fixed cases for malformed virus definition files that could trigger an invalid memory access in CSAgent.sys
Step 3: Clownstrike ships the buggy CSAgent.sys and it works for months without problems on customer computers with good virus definition files
Step 4: For some reason the webserver serving virus definition files to all endpoints started serving malformed virus definition files, sometimes they were all blank or filled with random bytes
Step 5: All clownstrike updaters now download malformed LE NEXT GEN APT PROTECTION CLOUD AI LIVE UPDATES into C:\Windows\System32\drivers\clownstrike
Step 6: CSAgent.sys reloads virus definition files
Step 7: CSAgent.sys parses them and crashes with PAGE_FAULT_IN_NONPAGED_AREA (in kernel that means memory at an oopsie address was accessed)
Step 8: Computer BSOD and reboots
Step 9: CSAgent.sys loads virus definition files
Step 10: Goto Step 7
The kernel driver was a ticking timebomb just waiting for a bug in the CDN. I think it was some funny caching bug with le cloud http loadbalancer. Users reported that their C-00000291-00000000-00000032.sys contained random shit and actual real files that were a completely different part of the software, like US localization files.
You can see it in the diff between CSAgent.sys_18511.sys and CSAgent.sys_18513.sys, they changed size checks and increased sizes of buffers so that future malformed virus definition files wouldnt crash.
That sounds like some virus protection on the server was stopping the reads from disk, and instead of throwing an error it was providing 0's for the output data instead.
It'll be funny if there's actually some antivirus package that caused it. ;)
Basically the linker couldn't open the file on windows (because it was locked by another process scanning it), and didn't error. Just replaced the object code to be linked with zeros.
People couldn't figure out what was wrong until they opened a debugger and saw large chunks of object code replaced with zeros.
the flawed data was added in a post-processing step of the configuration update, which is after it's been tested internally but before it's copied to their update servers
zero_output_file(fh, len(file))
flush()
fill_output_file(fh, data)
All of that is secondary: they wrote and shipped code that blindly loaded and tried to parse content from the network, and crashed when that failed. In kernel mode.
Honestly it’s probably good that this happened, because presumably someone malicious could use this level of broken logic to compromise kernel space.
Certainly the trust they put in the safety of parsing content downloaded from the internet makes me wonder about the correctness of their code for reading data from userspace.
Probably UCS-2/UTF-16 encoding with ascii data.
I'm reminded of how you have to use `{empty}` character refs in lightweight markup like Asciidoc to "hold your place" in a list, in case you need the first element in the list to be an admonition or block. Like so:
. {empty}
+
WARNING: Don't catch yourself on fire!
+
Pour the gasoline.
<procedure>
<step>
<warning> Don't catch yourself on fire!</warning>
<para>Pour the gasoline.</para>
</step>
</procedure>
Anyways, I'm spitballing that these old-timey nulls I'm seeing are being employed in an equivalent way, some sort of internal bespoke workaround to a format restriction.
Their standard methodology was to deploy untested stuff.
Think about it this way. At the present time, CrowdStrike can accept dark money from an entity in order to halt the world's compute capacity in a plausibly deniable way. When the dust settles, they will be found guilty of negligence at most. This will allow this perverse incentive to persist.
If the stakes for bringing down a great deal of American compute infrastructure were greater, this would happen far less often. It is simple economics of risk.
Furthermore, CrowdStrike, being an entity that enjoys access to privileged sales channels thanks to its relationship with the government, is exactly the sort of entity that should be viewed as skeptically as possible, if the interests of the US Government are paramount. Why should the US Government create a small monopoly for CrowdStrike and not expect the strictest controls? That is not only bizarre, it is stupid.
Hope for the best and plan for the worst. That is how an entity should be viewed when it becomes critical to national security.
Which nation exactly? Who on earth "wins" by crashing vast numbers of PCs worldwide?
Many of the potential foes you might be thinking of are unlikely to actually run CS locally but its bad for business if your prey can't even boot their PCs and infra so you can scam them.
I might allow for a bunch of old school nihilists getting off on this sort of rubbish but it won't last and now an entire class of security software, standards and procedures will be fixed up. This is no deliberate "killer blow".
Who knew that well meaning security software running in Ring 0 could fuck up big style if QA takes a long walk off a short plank? Oh, anyone who worked in IT during the '90s and '00s! I remember Sophos and McAfee (now Trellix) and probably others managing to do something similar, back in the day.
Mono-cultures are subject to pretty catastrophic failures, by definition. If you go all in with the same thing as everyone else then if they sneeze, you will catch the 'flu too.
Anyone who needs a big distraction so he can more likely achieve his real objective unnoticed.
Fair enough but if you are going to take over the world you don't crash everything, indiscriminately unless you are a really sad old school nihilist.
This is not subtle. It is so stupid that the only explanation is stupidity or a silly experiment.
The first thing I did is start to analyze my logs, just in case. I don't use CS but therefore by the Grace of God, go I!
If you have a particular target, you don't fuck up the entire world.
We will probably have to wait for CS' own report.
I'm at the stage where I'm thinking "%$#@ this, I'm never going to write to the Windows file system again, I'm just going to create an SQLite DB and write to that instead". Will that fix my problems?
Most of the files that are getting corrupted are being written to in an append-only fashion, which is generally one of the mechanisms for writing to files to avoid corruption, at least on POSIX.
I am not saying it happens often, but it does happen once in a while.
The file was corrupted, but the file-system remained consistent.
Sounds like you have an encoding issue somewhere, windows has it's own charset - Windows-1252, so I'd vet all your libs that touch the file (including eg. .Net libs etc). If one of them defaults to that encoding you may get it either mislabelling the file encoding, or adding in null after each append etc.
SQLite is tested cross-platform so 100% the file will be cross-platform compatible.
* It's standard for "security" software to be installed and hook into the filesystem. Normally this only means giving up on several orders of magnitude of performance, but as this incident reminds us: security software is indistinguishable from malware.
* The system has a very long history of relying on exclusive access to files. This is one of the major reasons for Windows needing to restart so many times, but also leads to a lot of programs being badly written, among other things making false assumptions regarding atomicity in the presence of system failure.
* There's a difference between text files and binary files, and thus a whole pain of adding/removing \r before \n (which among other things means byte counts are wrong and checksums mismatching).
I'm curious how this scenario, malformed definitions file signed by CS itself is not verified in any way on the endpoint. Like a try {} catch something...
I mean, the failure is spectacular in its simplicity.
as soon as security can come fully uncoupled from physical competence and the factual constraints of engineering, it will get subsumed into the managerialist hivemind as a rhetorical lever on narrative actuated cogs, and become as meaningless as everything else they do in service to administrative power for its own sake. I have a real problem with these people. :)
This crap shouldn't be run in kernel space. But putting that aside, we need kernels that will be resilient to and reject this stuff.
I don't like what Microsoft is doing with Windows and only use it for gaming (I'm glad Linux is becoming a good option for that), so I'm far from being a "Microsoft fan", but Windows is very good at installing the software needed. Plug a GPU, mouse, etc, from any well known brand and it should work without you doing much.
I didn't have to install anything on my Windows PC (or my MBP) last time I bought a new printer (Epson). The option to let Windows install the drivers needed is enabled though... some people disable that.
Disagree. It is everyone's fault. It is CrowdStrike's fault for not testing their product. It is Microsoft's fault for allowing CrowdStrike to mess with kernel and not vetting such critical third parties. It is the end customers' fault for installing crapware and not vetting the vendor.
CloudStrike is effectively running as a Windows kernel module. Which in Windows, might as well be the Windows kernel itself. There should be a deliberate difference between things that a bare operating system needs to function vs. things which are allowed to execute in a privileged manner.
This isn't a restrictive operating system. You have to trust your operating system vendor just a little more than you trust the 3rd party software installed on it.
My main machine is a Macbook Pro and one thing that annoys me a lot is the way Apple handles apps that are not notarized. I don't use iPhones because of the system restrictions (file access, background running, etc) and because I can only install what Apple allows on their store. You can see why I don't want Microsoft to hold my hand when I use Windows... it's my machine, I paid for it, I should be able to install crapware and extend the system functionality if that's what I want especially when I pick an OS that allows me to do that.
In this case, enterprise customers decided to use an OS that allows them to also use CrowdStrike. Maybe Microsoft could handle this stuff better and not show a BSOD? I guess so, but I won't blame them for allowing these tools to exist.
Don't get me wrong, there's a place for very restrictive operating systems like iOS or ChromeOS, but they're not for everyone or enough for all tasks. Windows is a very capable OS, certainly not the best option for everyone, but the day Microsoft cripples Windows like that, it's the day I am forced to stop using it.
How would the OS differentiate between a compromised agent and a faulty one ?. Do you want the OS to continue boot when it is potentially compromised ? That is likely to cause a lot more damage than yesterday did.
Maybe Microsoft doesn't offer an ABI or whatever other access is needed for the CS module. So there's some work that needs to be done on the kernel to enable the functionality to run in user space. Third-party libraries should not have privileged access to the kernel internals.
In this case, the CloudStrike kernel module attempted to read a bogus data file and crashed. The Windows kernel should detect the module crash, inform you in some meaningful way, but continue to boot. Maybe even in some limited execution model, but not a blue screen.
CloudStrike should have tested better. A LOT better. Multiple layers of tests better. They should most definitely take serious their obligations to the millions of customers that are affected by this. But Microsoft needs to own some of the responsibility too, by allowing a 3rd party kernel module to even be able to bring down its system. Just because the program was signed, doesn't mean when it crashes, Windows should crash with it.
Agreed, it's a Windows weakness. It should be improved. But if the system boots normally even if something like CrowdStrike fails, then you create other problems, which may be more serious than a BSOD.
I don't put the blame on Microsoft because enterprise customers should know how Windows works. Should we use Windows and some low level, 3rd party, remotely updated software (without a slow, controlled rollout) if you can't have BSODs? Yeah...
It's a big "cock up". I blame mostly CrowdStrike because they're the ones who messed up. Then we have those who decided to use Windows with CrowdStrike even on very sensitive systems. Microsoft is at fault for creating an OS with this weakness, but this is how Windows works... if it's not good enough for the task, don't use it.
It's Microsoft's fault that someone wrote drivers for its OS?
Apple won't let you add an app to their app store without a vetting process. Microsoft lets you alter the kernel without a vetting process? How many people died today because they didn't get surgeries and because of failed 911 calls?
Microsoft is not recommending to run software that monkey patches kernel for mission critical applications or run them on public networks like this. Does your car download software like this and crash on the road because of faulty update?
Time and again we keep seeing down times at hospitals due to ransomware or hacks, when will these institutions that people trust their lives with, take their IT posture seriously rather than blame anyone else.
Also, label printers, why the heck does zebra only do EPL or ZPL? Why not pcl6 or PS like the rest of the universe?
The point is that printers are bullshit. Nobody knows how they work, and assuming that microsoft should just figure it out on its own is at least in my opinion, unreasonable.
Also, I'm glad Microsoft doesn't provide an easy way to get what is essentially complete control over a machine, and every single event/connection/process that it has.
And my systems are all halfass, and I don't really know what I'm doing. I can't imagine actual real professionals letting that moulder its way downstream. Maybe their stuff is just way more complex and amazing than I can possibly imagine.