Unfortunately it's rather lacking in both technicality and detail.
Give them a minute.
We have to legislate out big tech and start holding them accountable. Prison time for execs and significant fines is the correct and effective method.
There's a long list of incredibly damaging fuckups by software companies, but I can't think of a single example of an existence-ending judgement from litigation.
The biggest reason why billion-dollar software companies continue to release shitty and dangerous products is because they face zero legal liability for their negligence.
If Crowdstrike is litigated into non-existence, it will not be in America, because the law doesn't work like that. Worth noting that there are a lot of software companies in America. This is not a coincidence.
Almost every large tech company in existence has had some sort of fuck up and survived. (Intel CPU vulnerabilities, iCloud security scandal, Sony multiple data leaks, ..).
CISOs buy EDR. In order to kill CrowdStrike, you'd need competitors with similar capabilities who haven't caused similar but smaller and less publicized outages or performance hits (off the top of my head, Tanium and Carbon Black have. I was there.) And that haven't been publicly hacked due to equally boneheaded issues in other products recently (like Palo Alto). So Microsoft... maybe.
Just one example - Fujitsu, associated with the biggest ever misscarriage of justice in England because of bugs in their Horizon software, still going strong...
https://en.wikipedia.org/wiki/Therac-25
No class action lawsuit, just some relatives of dead patients settling out of court.
Two years later it had 1 billion revenue.
I'd feel weird picking a protective app called "crowdstrike".
You don't go to Amazon for rainforest information.
You don't go to Google for information about big numbers.
You don't go to Uber for an Aryan vibe.
You don't go to Apple for fruit.
You don't open Windows to let in a cool breeze.
You don't need a trap for your wireless mice.
Monster isn't going to eat you.
Jeeves isn't actually a person you can ask, and knows less about buttling than you'd expect.
FireEye is competition to crowdstrike, not an STD.
You get used to it after awhile.
Now back to the topic.)
Greenland not being that green.
Flaming Saddles and all the names of restaurants serving some kind of spicy food, whose names emphasize the side effects.
It's not called Microsoft Ads 2020
I dont know if gibson took it or coined it in neuromancer but it at least makes sense as a name for a thing. This isn't the same as "random noun" for a company name.
Some notable quotes from Shawn Henry regarding this from that hearing:
>"There are times when we can see data exfiltrated, and we can say conclusively. But in this case it appears it was set up to be exfiltrated, but we just don’t have the evidence that says it actually left."
>"There’s not evidence that they were actually exfiltrated. There's circumstantial evidence but no evidence that they were actually exfiltrated."
>"There is circumstantial evidence that that data was exfiltrated off the network. … We didn't have a sensor in place that saw data leave. We said that the data left based on the circumstantial evidence. That was the conclusion that we made."
>"Sir, I was just trying to be factually accurate, that we didn't see the data leave, but we believe it left, based on what we saw."
>Asked directly if he could "unequivocally say" whether "it was or was not exfiltrated out of DNC," Henry told the committee: "I can't say based on that."
https://intelligence.house.gov/uploadedfiles/shawn_henry_tes...
.sys files are supposed to be protected system files and require specual privileges to touch. I imagine Crowdstrike requires some special type of Windows "root access" to operate effectively (like many antivirus packages) in order to detect and block low level attacks.
So where things likely went pear-shaped was Crowdstrike's QA process for config updates is possibly less stringent than core code updates. But because they were using .sys files it was actually given elevated privileges and gets installed during boot.
As for the actual bug, I expect it was either something like the sys file referencing itself or some sort of stack overflow somewhere, both of which I would pin on Microsoft for not being able to detect and recover from during boot up.
All of this is straight guesswork based solely on experience as a longtime Windows user.
Any file in C:\windows\ is protected by default and requires administrative privileges to modify. There isn't any extra protections for .sys files specifically.
>As for the actual bug, I expect it was either something like the sys file referencing itself or some sort of stack overflow somewhere, both of which I would pin on Microsoft for not being able to detect and recover from during boot up.
Since when is it the operating system's responsibility to recover from badly written kernel mode code?
My home PC's /boot drive keeps filling up with older kernels due to updates
In this case it’s a bit ironic since their whole business is mitigating risk.
Alternately, they did not ever test the ability of their code to accept arbitrary configurations. That totally could have been done, allowing fast config pushes. But it wasn’t. So as a reliability engineer, I’d point at that change: configs were originally expected to be tested, so the parser was only tested to be sure it showed issues on pre-release testing. Later, they started shipping configs faster than their release cycle, but didn’t check to requalify the parser for this new requirement.
Behind that, I’d look at the engineering and product culture that had that happen. Is there a list of what’s been tested for what purpose? Was the expectation that configs get tested like software written down someplace, and do the roles who scheduled that config release read that place? Who is organizationally set up to notice this, and why is it a director-level IC straight out of xkcd 2347?
Why would any of them come forward right now?
if err != nil { return err }
I hope no one is dumb enough to spill the beans just for some karma. Especially when you don’t know what the official findings are yet.
Sounds like a great swing trade to take a long position in CRWD for retail.
I guess it would depend how honest they are in their communication. Presumably if they outright lied, it would give enough incentive for someone to create an throwaway account via a VPN from a browser and machine instance they never used to say something like “hell, no, that’s not what happened”
Who are we to make this demand? Most likely technologists, managers, specialists, and concerned citizens with the expertise and insight to recognize the dangers inherent in our increasingly careless approach to ... many things, but, particularly technology. Who is to uphold the standards that ensure the safety, reliability, and integrity of the systems that underpin modern life? Government?
Historically, the call for accountability and excellence is not new. From Socrates to the industrial revolutions, humanity has periodically grappled with the balance between progress and prudence. People have seen - and complained about - life going to hell, downhill, fast, in a hand basket without brakes since at least Socrates.-
Yet, today’s technological failures have unprecedented potential for harm. The Crowdsource outage killed, halted businesses, and posed serious risks to safety—consequences that were almost unthinkable in previous eras. This isn't merely a technical failure; it’s a societal one, revealing a disregard for foundational principles of quality and responsibility. Craftsmanship. Care and pride in one's work.-
Part of the problem lies in the systemic undervaluation of excellence. In pursuit of speed and profit uber alles. Many companies have forsaken rigorous testing, comprehensive risk assessments, and robust security measures. The very basics of engineering discipline—redundancy, fault tolerance, and continuous improvement—are being sacrificed. This negligence is not just unprofessional; it’s dangerous. As this outage has shown, the repercussions are not confined to the digital realm but spill over into the physical world, affecting real lives. As it always has. But never before have the actions of so few "perennial interns" affected so many.-
This is a clarion call for all of us with the knowledge and passion to stand up and insist on change. Holding companies accountable, beginning with those directly responsible for the most recent failures.-
Yet, it must go beyond punitive measures. We need a cultural shift that re-emphasizes the value of craftsmanship in technology. Educational institutions, professional organizations, and regulatory bodies must collaborate to instill and enforce higher standards. Otherwise, lacking that, we must enforce them ourselves. Even if we only reach ourselves in that commitment.-
Perhaps we need more interdisciplinary dialogue. Technological excellence does not exist in a vacuum. It requires input from ethical philosophers, sociologists, legal experts. Anybody willing and able to think these things through.-
The ramifications of neglecting these responsibilities are clear and severe. The fallout from technological failures can be catastrophic, extending well beyond financial losses to endanger lives and societal stability. We must therefore approach our work with the gravity it deserves, understanding that excellence is not an optional extra but an essential quality sine qua non in certain fields.-
We really need to make this be an actual tuning point, and not just another Wikipedia page.-
And companies and their executives like these walk away.
Can't blame anyone for not caring anything.
Feed it to an LLM to summarize /s
"This is an article about a Reddit thread discussing why a diesel repair shop had a system-wide failure. The conversation centers around why industrial equipment, like lifts and cranes, would be running Windows and be connected to a network. People debate the pros and cons of having internet-connected machinery, with some arguing for the benefits of remote monitoring and updates and others expressing concerns about security vulnerabilities. Many users point out that some critical systems, like elevators, could be controlled with simpler and more reliable solutions."
Perfect example: I own a bunch of iHome smart outlets that I use with Apple HomeKit. Got a whole box of the things off ebay years ago, paid about $6 each. Along with (obviously) being a smart home accessory that runs via HomeKit, they have a little button on the side you can push to toggle the power on and off. So, in the odd event that my network is being uncooperative and I need to turn a light off, I can do that very easily. Not as easily as I could with my phone, but still quite easily.
I don't understand why industry is so hostile to this. It just makes sense to me: build in all the smart features you like, but if they're not working, just have the bloody thing operate like a normal... whatever. Microwave, lamp, etc.
All too true. But between the "we can sell far more stuff if we convince people to replace things that still work fine" of Modern Capitalism, and the "I can brag about all our Shiny New Stuff, and it ain't my money" of Modern Executives...
The thread on Hacker News revolves around a significant incident where a CrowdStrike update led to widespread issues, including Windows blue screens and boot loops. This event affected various industries, including facilities where heavy machinery like lifts and cranes depend on Windows-operated systems.
1. *Incident Overview*: The CrowdStrike update caused critical system failures, leading to operational disruptions across multiple sectors. Users reported being unable to operate essential equipment, disarm alarms, or use communication tools, which brought businesses to a standstill.
2. *Root Cause Analysis*: The primary issue stemmed from an update pushed by CrowdStrike that conflicted with existing systems, particularly those running on Windows. This led to blue screens and boot loops, effectively rendering systems inoperable. The reliance on Windows for industrial control systems (HMI - Human-Machine Interface) exacerbated the impact.
3. *Quality Assurance and Deployment*: There was significant discussion on how such a critical update passed QA. It's suggested that the widespread deployment of uniform security policies without tailored considerations for different system requirements contributed to the problem. In some cases, security software was installed on systems where it might not have been necessary, driven by compliance needs and centralized IT policies.
4. *Implications and Lessons*: The incident highlights the vulnerabilities of interconnected systems and the challenges in managing updates and security across diverse operational environments. It underscores the need for robust QA processes and the potential risks of over-centralized security policies.
For more detailed discussions and perspectives, you can review the full thread [here](https://news.ycombinator.com/item?id=41002195).
I wonder if this is a preventable problem, hmmmm.
No sir, not at all! It was inevitable! Let's just leave this behind us! Those who think it's preventable are part of a certain "evangelism strike force", I tell you! They make zero good arguments, never listen to them!
Back to business as usual, boys! C/C++ are without a single flaw!
Apple has a vetting process before they will allow an app to be added to their app store. Why doesn't Microsoft have a vetting process before allowing a third party to mess with the Windows kernel? Does Crowdstrike have SOC2 or some other certification to make sure they are following secure practices, with third-party verification that they are following their documented practices? If not, why not? Why doesn't Microsoft require that?
It is clear that the status quo can't continue. Think about the 911 calls that didn't get answered and the surgeries that had to be postponed. How many people lost their lives because of this? How does the industry make sure this doesn't happen again? Just rely on Crowdstrike to get their act together? Is it enough to trust them to do so?
https://blogs.windows.com/windowsexperience/2018/12/19/drive...
That didn’t happen. Apparently, the policy for drivers does not apply to kernel modules… WTF!!
No operating system can guarantee that a driver will never cause the machine to crash. This wasn’t Microsoft’s fault.
That leaves you either not responding quickly or responding with uncertified updates. In the past, we have examples of not responding quickly that took down large chunks of the internet (I don't remember the examples, but they were quite famous at the time). Now we have an example of a fast, uncertified update taking down a large chunk of the internet.
So, given that it can take down much of the internet no matter which we choose, now what do we do?
What can we do?
Require them to have documented processes, and require periodic (like every 6 months) third-party auditing that they have the right processes, and they are complying with their own processes.
More info: https://en.wikipedia.org/wiki/System_and_Organization_Contro...
Nearly all banks have long long lists of certification, they still have extremely bad customer-side security processes because you can "interpret" various guidanecs and pay the right auditors enough to have it ignored.
Auditing every data file update seems just as error/system failure prone as Crowdstrike's process was. I don't see a clear reason why Microsoft would have any better incentive than Crowdstrike here.
I do think that maybe the commercial OS vendor has _some_ support responsibilities to at least warn and discourage customers from using the product in dangerous ways? I mean, it's not like we're talking about a couple people installing bad kernel drivers here, we're talking about a worldwide incident. WHQL seems like an admission that Microsoft knows they need to keep dangerous drivers out of the ecosystem.
Imagine if malware could somehow crash this module - would you be happy about the OS automatically rolling bank introduction of said module, opening your system to vulnerabilities?
I suspect the US government might have pressured them to push this update because they’ve found out that Crowdstrike’s system has either already been breached or has a significant zero-day security vulnerability.
Now, taking off my tin foil hat…
But I 100% agree. There is a good book on that topic: Normal Accidents: Living with High-Risk Technologies by Charles Perrow
I doubt there's much they could do about forcible government intervention.
Now let me give you the realistic news: every executive everywhere will just hand-wave this away with "it will not happen again anytime soon, or ever, right?" and be back to business as usual.
The extreme cost-cutting and allowing non-technical people to rule us is what is getting us into these messes every time. And humanity in its totality seems almost completely unable to learn.
CrowdStrike's ability to make sales is (I assume) very dependent on their appearance of trustworthiness.
Given the microscope they're under, and how serious some of their customers are, I think they'll need to make a convincing case that this kind of problem won't happen again.
Reminds me a little of Boeing, now that I think of it.
I can accept your POV of me being too cynical btw. But I think it's also important to give you one extra nuance: everybody feels awkward after "accidents" like these and everybody is very willing to accept half-arsed explanations (that usually have zero details or guarantees of future better practices) and they just want to move on and forget about it.
Historically it seems that this approach wins in 99% of the cases.
And, if the various stories are to believed, removing or neutering the execs and managers that required an update be pushed out immediately without passing through the normal staging system.
Cant follow either of you XD
The size of the organization may only serve to increase the number of potentially bad pieces of code from just plain old tech debt, poorly engineered, or just a hack job implementation. The ability to control quality of code is highly dependent on how staunch the organizational culture is around keeping the repos clean of dead, kludgey, or just bad code, and strict enforcement of agreed upon architecture.
Why would they ask to put half the country to a standstill because a foreign threat actor might have control of the system and might put half the country to a standstill?
It’s an action plot from a movie. Improbable, but not impossible. I don’t think our government is nearly that competent, but who knows.
The simpler explanation is often the answer here. Either (or both) their internal QA and release process is flawed or specific employees did not follow the process correctly.
These fantastical explanations of some big government conspiracy only serve to make the believers feel better about the fact that nobody is actually in control and shit just happens randomly which is honestly a far more terrifying prospect.