CrowdStrike debacle provides road map of American vulnerabilities to adversaries
nytimes.com
nytimes.com
Security is not a feature that can be layered on. It has to be built in. We now have an entire industry dedicated to trying to layer security onto Windows -- but it still doesn't work.
1. How does the software obtain new data at run time? 2. How do you make sure that thing doesn't pose a security hole when a vulnerability gets discovered? (assuming this never happens is unrealistic)
I agree with you but I also want to play the devil's advocate: using software like CrowdStrike is not what I would call being "cloud-reliant". It's simply using highly-privileged software that appears to have the ability to update itself. And that is likely far more common than cloud-reliant setups.
I suppose you could build it as a UEFI module that relies on the UEFI firmware to initialize the hardware but then you get a text only interface. But then the UEFI is the OS.
But this outage was not an OS problem. It was an application bug that used invalid pointers. If it was a unikernel it still would have crashed.
from the reporting so far, no one has died as a result of the Crowdstrike botch. For my money, that sounds like it's not being used in 'critical industry'.
/unset
There were several 911 service outages included in the news yesterday, so I would definitely say agree those fall into the category. I haven't seen how many hospitals were deeply affected; I know there were several reports of facilities that were deferring any elective procedures.
I would imagine it’s the same story at human hospitals too that ran afoul of this. I wouldn’t expect life-critical systems to go offline, but there’s many other more mundane systems that also need to function.
The vendor who makes the software has always written for Windows (or in reality, wrote for either DOS or OS/2 then transitioned to NT4). History, momentum, familiarity, cost, and ease of support all are factors (among others, I'm sure).
Security is a process, not a product.
And yes, distros require frequent updates, though more to your point, you can limit the scope of installed software. I'm sure airport displays don't need MPEG2, VP1 and so on codecs, for instance.
It's also important to remember that there is a lot of 'garageware' out there with these specialized systems. Want SAML/OIDC support? We only support LDAP over cleartext, or Active Directory at best. Want the latest and greatest version of Apache Tomcat? Sorry, the vendor doesn't know how to troubleshoot either, so they only "support" a three year old vulnerable version.
Ran into that more than a few times.
Given the hypothesis of what caused the BSOD with Crowdstrike (NUL pointer), using a safe language would have been appropriate -- it's fairly easy in this case to lay the blame with CS.
Microsoft supplies the shotgun. It's the vendors responsibility to point it away from themselves.
They don't, until the day the airport managers are approached by an advertising company waving the wads of cash the airport could be 'earning' if only they let "AdCo" display, in the top 1/4 of each screen, a video advertising loop. At which point, those displays need the codecs for "AdCo's" video ads.
Security is a process, not a product...
The vendor who makes the software has always written for Windows (or in reality, wrote for either DOS or OS/2 then transitioned to NT4). History, momentum, familiarity, cost, and ease of support all are factors (among others, I'm sure)...
That's starting the argument with "weight loss is about overall diet process, not individual choices" and then hopping to "ice cream for dinner is good 'cause it's convenient and I like it".
The statement "Security is a process, not a product." means you avoid shitty choices everywhere, not you make whatever choices are convenient, try to patch the holes with a ... product ... and also add an extra process to deal with the failures of that product.
They want to hear that they can pay $X dollars to this service provider, and tick all of the cover-your-ass boxes in the security checklist; where $X is the cheapest option that fits the bill.
That's the negative part. The positive part is that security considerations have to run through an entire organization because every part of the organization is an "attack surface".
The whole concept of CrowdStrike is that it's there to prevent individual users from doing bad things. But that leaves the problem of CrowdStrike doing bad things. The aim of security as process is avoiding the "what-a-mole" situation that this kind of thinking produces.
Operating systems that don't require frequent security patches aren't profitable.
Anyway, this is the step of late-phase capitalism that comes after enshittification. Ghost in the Shell 2045 calls it "sustainable war". I'd link to an article, but they're all full of spoilers in the first paragraph.
It probably suffices to say that the series refers to it as capitalism in its most elegant form: It is an economic device that can continue to function without any external inputs, and it has some sort of self-regulatory property that means the collateral damage it causes is just below the threshold where society collapses.
In the case of Cloud Strike, the body count is low enough, and plausible deniability is low enough that the government can get away with not jailing anyone.
Instead, the event will increase the money spent on security theater, and probably lead to a new regulatory framework that leads to yet-another layer of mandatory buggy security crapware (which Cloud Strike apparently is).
In turn, that'll lower the margins of anyone that uses computers in the US by something like 0.1%, and that wealth will be transferred into the industry segment responsible for the debacle in the first place. Ideally, the next layer of garbage will have a bigger blast radius, allowing the computer security complex to siphon additional margins.
Consider the reasons people need this endlessly updated layer of garbage, as you put it. The constant evolution of 0-days and ransomware.
I'm a developer, and also a sysadmin. Do you think I love keeping servers up to the latest versions of every package where a security notice shows up, and then patching whatever that breaks in my code? I get paid for it, but I hate it. However, the need to do that is not a result of "late-stage capitalism" or "enshittification" providing me with convenient cover to charge customers for useless updates. It's a necessary response to constantly evolving security threats that percolate through kernels, languages, package managers, until they hit my software and I either update or risk running vulnerable code on my customers' servers.
That's not strictly true, but it's true in an economic sense:
You could just move your servers to OpenBSD, and choose to write software that runs on top of its default installation. There have been no remotely exploitable zero days in that stack for what, two decades now? You could spend the time you currently use screwing with patches to architect the software that you're writing so that it's also secure, and so that you could sustainably provide more value to whoever is paying you with less effort.
Of course, the result wouldn't never obtain FIPS, PCI, or SOC-2 compliance, so they wouldn't be able to sell it to the military, process credit cards, or transitively sell it to anyone that's paid for SOC-2 compliance.
Therefore, they can either have something that's stable and doesn't involve a raft of zero days, or they can have something that's legally allowed to be deployed in places that need those things. Crucially, they cannot have both at the same time.
Over time, an increasing fraction of our jobs will be doing nothing of value. It'll make sense to outsource those tasks, and the work will mostly go to companies that lobby for more regulatory capture.
Those companies probably aren't colluding as part of some grand conspiracy.
It's also in their best interest to force people to use their stuff. Therefore, as long as everyone acts rationally (and "amateurs" don't screw it up -- which is a theme in the show), the system is sustainable.
Incredible how easy it was to prove this wrong in less than 5 minutes.
Most people know how to use a windows computer.
Most IT desktop support knows how to use and manage windows. Even building facilities folks can help support them.
Microsoft makes it easy to manage a fleet of computers. They also provide first party (along with thousands of 3rd parties) training and certifications for it.
Windows are the de facto Business Machines.
Most signage companies use windows.
Finding someone who knows a BSD is not easy.
You would send an engineer into an airport to reboot a sign?
At others, airline staff is responsible for it. And just like airport staff, a tech who can deal with Firefox on Windows is cheaper than someone who can troubleshoot the same in Linux or a more custom system.
Another take to be done here is: computers shouldn't have unfiltered internet access all the time.
Whitelist it and once every 3 days open the internet gates.
(Easier said than done)
A windows computer that relies on cloud services, as an increasing and often nonsensical subset of the functionality on one does, can often only be fixed by Microsoft directly
Microsoft intervenes directly and spends billions of dollars annually on anticompetitive tactics to ensure that other options are not considered by businesses
And with this monopoly, it has shielded itself from having to compete on even crucial dimensions like reliability, maintainability, or security
I know of a very small airport where what is displayed over the HDMI part is essentially Firefox at fullscreen with powersaving disabled so the screen does not blank. Some of them are Intel NUC, some of them are Raspberry Pi with HSM in a box. These devices basically "boot to Firefox" with relevant credentials read off internal TPM/HSM.
Those among airport staff who do not know how to use a computer at all can get them working by just plugging them in.
> Most people know how to use a windows computer.
They know enough to open a browser.
> Most IT desktop support knows how to use and manage windows.
They know how to cope with Windows, at best.
> Finding someone who knows a BSD is not easy.
BSD is everywhere and in far more places than Windows, like almost every car sold after 2014. But you never ever see BSD because it's already-working with nothing for the end customer to do.
Airport staff are not debugging the windows install. They power-cycle it and see what happens, otherwise call the vendor to come in.
So there's no actual reason other than lazyness to build kiosk mode computers on windows.
Or even ChromeOS which has insane security.
> but it still doesn't work.
It works momentarily but there will always be 0-days the people who make the exploits intimately know the windows API internals.
ChromeOS is a Linux distro BTW
There's an entire industry for guard-railing LLMs now. Go figure.
This is a common misunderstanding, an OS that receives frequent security updates is a very good thing. That means attention is being paid to issues being raised, and risks are being mitigated. Security is not a 'checkbox' it's more of a neverending process because the environment is always in a state of flux.
So to flip it, if an OS is not receiving updates, or not being updated frequently, that's not great.
What you want is updates that don't destabilize an OS, and behind that is a huge history and layers of decisions at each 'shop' that runs these machines.
Security is meant to be in layers and needs to be built in.
> but it still doesn't work.
It does work because the 'scene' has been silent for so long, but what we as humans notice is the incident where it didn't.
That's before even addressing mistakes.
You could get a Solarwinds type situation where the adversary has the signing keys and ability to publish to the website.
You might also find that the vendor ships a library (like libxz) as a part of their invisible or hidden supply chain, that is able to be compromised.
You might find that one of the people working at the company makes a change to the code to enable remote access by the adversary in a targeted collaboration/attack.
The problem isn't that signing key (although I could delve into the lengths you'd need to go to to keep that secret under these threat models) - the problem is what they sign. A signed end release binary or series of packages isn't going to address the software source code itself having something added, or the dependencies of it being compromised.
Not patching is insane -- you'll let script kiddies in. Patching might not stop the next Stuxnet author, but you'll slow them down _and_ have fewer script kiddies.
A lot of people seem to be focusing on how the band-aid of automatic security updates can be ugly without considering the hemorrhaging that it's actually stemming. Nobody's stepping up with a realistic solution to the problem, which means we're stuck with the band-aids.
[1] https://www.usenix.org/system/files/1401_08-12_mickens.pdf
We've got a bunch of computers that mostly don't make mistakes at the hardware layer. On top of that, we can write any programs we want. Even though the halting problem exists, and is true for arbitrary programs, we know how to prove all sorts of useful security properties over restricted sets of of programs.
Any software security pitch that starts with "when the software starts acting outside of its spec, we have the system ..." is nonsense. In practice, "acting outside its spec" is functionally equivalent to "suffers a security breach".
Ideally, you'd use an operating system that has frequent updates that expand functionality, that is regularly audited for security problems, and that only rarely needs to ship a security patch. OpenBSD comes to mind.
If software has frequent security updates over a long period of time, that implies that the authors of the system will continue to repeat the mistakes that led to the vulnerabilities in the first place.
How is that accomplished? Are OpenBSD programmers somehow vastly more competent, that they make security mistakes only 0.1% as often as other OS's?
I find that hard to believe. People are people.
> If software has frequent security updates over a long period of time, that implies that the authors of the system will continue to repeat the mistakes that led to the vulnerabilities in the first place.
Why would that be the case? Authors come and go, systems live on.
Security updates arise from a combination of auditing/testing and competence. 100 times as many security updates can arise simply because one OS is being used and battle-tested 100x more than another.
Nobody's smart enough to write code that "only rarely needs to ship a security patch". Not at the scope of an entire OS with thousands of people contributing to it.
Put simply, yes. If you read open OpenBSD's website what philosophies and practices drive how the OpenBSD project is run, you'll have an idea.
It makes it very hard to put any credence into your opinion when you are so judgemental with no information.
It's not about competence, it is about priorities.
OpenBSD obsesses about security, so that's what drives the decision-making.
All public companies are driven by profit above all, with the product being just a mechanism to get more profit. As a direct consequence, quality (and security, which is part of quality) is not the top priority. Security is only relevant to the extent its absence reduces profits (which very rarely happens).
TL;DW It isn't as much as people think.
Hardware, software and firmware are all prone to mistakes, errors and corner cases that are surprising. Security issues generally live in the intersection of systems with different metaphors. Hardware is not immune from issues, and software can help reduce that impedance mismatch.
[0] Found an instance here, no claim to its veracity or safety: https://www.gherush92.com/documents/744E.asp?type=2&file=C%3...
Minimal software and OS running on linux as a layer between any windows/whatever and internet connectivity. Minimize and control the exact information that gets to the less hardened and trustworthy/complicated computers
The kiosk display terminal is not something I care about that much.
> We now have an entire industry dedicated to trying to layer security onto Windows
Too bad we have no such layering in our networks, our internet connections, or in our authentication systems.
Thinking about it another way there's actually no specific system in place to ensure your pilot does not show up drunk. We don't give them breathalyzers before the flight. We absolutely could do this even without significant disruption to current operations.
We have no need to actually do this because we've layered so many other systems on top of your pilot that they all serve as redundant checks on their state of mind and current capabilities to safely conduct the flight. These checks are broader and tend to identify a wider range of issues anyways.
This type of thinking is entirely missing at the computer network and human usability layer.
There are more than two, and the vast majority of the time people don't need anywhere near the complexity that modern browsers have shoved into them. A lean browser that supported only a bare minimum of features would go a long way to reducing attack surface. As it is now, I already find myself disabling more and more functionality from my browsers (service workers, WebRTC, JS, SVG, webgl, PDF readers, prefetch, mathml, etc)
Nobody's commenting on that because it's the wrong thing to focus on.
1) This fuckup was on CrowdStrike's Falcon tool (basically a rootkit) bricking Windows due to a bad kernel driver they pushed out without proper hygiene, not on Windows's security patches being bad.
2) Linux also needs to get patches all the time to be secure (remember XZ?) It's not just magically secure by default because of the chubby penguin but is only as secure as it's most vulnerable component, and XZ proved it has a lot of components. I'd be scared if a long period goes by and I see no security patches being pushed to my OS. Modern software is complex and vulnerabilities are everywhere. No OS is ever bug-free and fully bullet proof in order to believe it can be secure without regular patches. Other than TempleOS of course.
The lesson is whichever OS you use, don't surrender your security to a single third party vendor who you now have to trust with the keys of your kingdom as that now becomes your single point of failure. Or if you do be sure you can sue them for the damages.
1) While CrowdStrike can be run on Linux it is less of a risk to use Linux without it than Windows. I don't think most Linux/BSD boxes would benefit from it. It could be useful for a Linux with remotely accessible software of questionable quality (or a desktop working with untrusted files) but this should not be the case for any critical system.
2) There is a difference between auto-updates (common in Windows world) and updates triggered manually only when it is necessary (and after testing in non-prod environment). Also while Linux is far from being bug-free, remotely exploitable vulnerabilities are rare.
Again, those auto updates that caused this issue were developed and pushed from Crowdstrike not from Windows. That tool does the same auto updates on Linux too. On Windows side you can have sys-admins delay Windows updates until they get tested in non-production instances, but again, this update was not pushed by Windows for sysadmins to be able to do anything about it.
EDR isn't antivirus. It logs and detects more than it prevents, and you need that on Linux as much as Windows. You can do incident response without it if you are shipping your logs somewhere, in the sense that you can do anything without any tool, but it's certainly a lot easier with.
Possibly you need it less than on Windows since it's easier (for now) to do kernel stuff with eBPF, but then somebody has to do the kernel stuff.
Speaking as a professional red teamer, no OS has a ton of RCE, but applications do, Linux applications no less than Windows ones. Applications aside I'd rather be up against Windows in the real world because of Active Directory and SMB and users that click stuff, but Linux running a usual array of Linux server stuff is OK too.
microkernels, microkernels, microkernels! https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...
It's just a likely they could crash a Linux machine by releasing an update to their Linux software that also referenced invalid memory.
Am I the only one that's seen drivers in Linux cause a kernel panic?
"Nobody ever gets fired for buying IBM" is as true as ever at least in the corporate world.
I was personally involved in a meeting where my firm's leadership advised a client who did fire their CTO and a bunch of other people for what was ultimately putting what they thought were smart career moves over their actual responsibilities.
Unfortunately, as you did just point out, the CEO, other execs, and board are often just as incompetent as the CTO/CISO who have such shit-brained mindset.
I understand the reasons for it, and why large, billion dollar companies try to create some sort of efficiency by centralising on one "vendor", but, then this happens.
I don't know how to fix the problem of following "Industry Trends" when every layer above me in the organisation is telling me not to spend the time (money) to investigate alternative software choices which don't fit into their nice box.
The same thing crash could happen with any kernel driver in any operating system.
You've never seen Linux crash because of a driver bug?
What makes you think so?
How is Linux better in that area?
Because in the non-Silicon-Valley world of software, if you pick Linux and it has issues, fingers will get pointed at you. If you pick Windows and it has issues, fingers will get pointed at Microsoft.
We moved to a more frequent update cycle because when a critical vulnerability was found, no one wanted to wait 6-12 months for the service pack.
Was there ever such a time? If so then tell me when it was.
"The latest chaos wasn’t caused by an adversary, but it provided a road map of American vulnerabilities at a critical moment."
I've no doubt that road maps of American vulnerabilities are currently being planned, roadmaped and stockpiled for future use by those who aren't on the best terms with the US.
In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
Remember, this happened big-time when Microsoft introduced Windows especially Windows 95. To capture the market Microsoft made everything as easy as possible for nontechnical users—just click on something and it'd happen, things would happen with ease. And all this happened without due consideration to security.
When viruses, vulnerabilities, breaches got out of hand restrictions were introduced which meant users had less freedom to do what they'd gotten used to doing. What Microsoft did was to get the world used to slack operating procedures and efforts reign this in has met with user resistance ever since.
We're now stuck with a major problem that was easily foreseeable even before Microsoft launched Windows 95. Fixing it will be extremely difficult.
There were years of cyberattacks against pretty much every peice of critical infrastructure they have. Things went down, there were disruptions, but they adapted. Sometimes by falling back to low-tech solutions, sometimes by developing new systems with robustness into new systems and purging the old (much easier to politically justify when the problem is tangible and immediate).
I seem to recall that one of the first things we did when tensions started ramping up was sending teams of cyber security experts from the NSA to help them lock down and root out infiltrations.
My first thought in all this was wondering if there's a business opportunity for a consulting firm or startup that designs and manages offline paper backup systems that can quickly and seamlessly integrate back with digital systems once they come back online.
Fully comprehensive, regularly trained manual operations are very expensive to develop and test. Only the most safety-critical organizations will be able to justify and have the resources to effectively implement them. Air-traffic control, hospitals, nuclear plants, etc. And, they already have done it.
It's not offline paper backups that are needed but rather the reverse—offline paper-based systems used as masters!
The heart of any critical infrastructure—specifically the part of the heart that's the most vulnerable is comparatively small compared to the large masses of ancillary data and thus could be managed on a paper-based database (as they once were before computers).
With computers and IT infrastructure as they're currently implemented—not as computer science says they ought to be—a secure filing cabinet/paper-based database is much more secure than an ephemeral one that has no physical or volumetric presence and which takes precious little to shove it from one end of the planet to the other. The caveat is of course, the database must be secured against physical access and located in a secure building, etc.
Let me state why. Comparatively speaking, in recent times there are very few secure bank vaults and such that criminals have breached. The number is so small I can't remember when I'd last heard of a bank vault robbery. Another way of looking at it is to ask yourself when was gold last stolen from Fort Knox or the Bank of England, or $100 bills stolen from the US treasury/mint before their distribution.
Why so few robberies you may well ask. We've had hundreds of years of experience locking up these valuables and although the current systems used to secure them aren't watertight and likely never will be, they're nevertheless sufficiently secure to the extent that the few breaches that do occur from time to time are manageable. With physical security, we've found a workable balance between security and workability.
With the few robberies that occur it's not worth the effort of tightening security further, to do so would not only add considerably to the cost but also physical access would be more difficult thus less convenient to use because of the additional protocols that would have to be put in place to reach the higher security level.
Also, think for a moment that if you could gain access to a secured paper-based database how quickly could you copy it, and how would you copy it? Right, both would be very difficult. On the other hand once an electronic database has been breached megabytes if not gigabytes can be sucked out within seconds.
In practice, the electronic/digital world has nothing as 'bulletproof' as a physically secure system. Given the statistics—the rate of cyber breaches, personal data stolen, Bitcoin thefts, etc., etc. that occur not on a yearly but rather on a daily basis, one simply can't argue that collectively IT/electronic systems are more secure than physical, paper-based ones.
Back to the physical database: a secure paper-based database would always be offline, if some data are needed from it then they have to be extracted manually, then vetted and encrypted before being put on line (that's if it's actually necessary to put highly secure stuff on line at all).
As things stand, owners of information have a choice, store it in an electronic system and take advantage of the operational advantages that it offers or use secure paper-based storage and suffer the inconvenience. One can't have it both ways.
The reason why we've so may data breaches is that the average punter far prefers electronic data systems for their convenience. On evidence, convenience is seen as more important, in practice its value far outweighs data security and integrity.
Something else that would be a challenge for your idea would be how to handle the orders of magnitude efficiencies in scaling gained from digital technology.
Emergency call centres often (or at least should!!) have these paper or card based backup processes - notes are taken on the card based on the call, addresses written manually, and the card is carried to a dispatch desk (perhaps sitting on a handheld radio if the main system is unavailable), passing information to response teams. Handling of each call requires more people, and gets you a much lower throughput (manually writing addresses, without lookups for spelling correction, reading them with phonetics over the radio to drivers etc).
How many times have you tried to call a business during an incident or disruption and been unable to get through on the phone, because they aren't staffed to a level that can handle any significant % of their customers calling at once? (Often, these companies lack tech company style realtime status pages as well, which could arguably reduce call numbers).
I do think there's some merit in trying to help organisations improve process and procedure resilience, but it doesn't strike me that it will be effective unless normal staffing levels are nearer the levels needed for "crunch" operations (or people are kept "on call" at extra cost to be available).
There are however a lot of good lessons that should be learned from the wider fiasco around technology resilience and systems design, and part of that should include independent (with as close to entirely independent failure modes as possible) redundancy systems.
Are they unique in any way? Or is it just yet _another_ case of Windows software being deployed in critical roles and basic 0day vulnerabilities and exploits being applied against it?
If so.. the lesson has been known for decades.
> sending teams of cyber security experts from the NSA
It's nice to know our security agencies have time for games of whack a mole.
It was a goal for a long time, and I'd say we use to be more resilient pre-cloud SaaS auto-update everything. When every software solution installation is on private networks, with fundamentally different architectures (both machine and topology), along with a wide selection of even very poor quality software, was a lot more resilient than what we have today.
Today a single outage in a single service (say AWS) can grind a large number of companies to a halt. A bad update like this one immediately impacts everyone all at once and has a domino effect. That didn't use to happen.
We've been concentrating our collective architecture into a few best practice tools but that all become single points of failure for not only digital attacks, but misconfigurations, mismanagement, company failures, exhausted underpaid engineers, optimizations, etc.
> Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
This isn't necessarily true, and I'd argue quite the opposite direction has been happening in the security industry over the past decade or so. People realized that hard security would only cause users to find simple predictable bypasses that would overall _weaken_ the security posture. You just have to look at the evolution of NIST recommendations around passwords to see this happening.
Must change a password every 90 days that can't be the same as your last 10 passwords and complex password requirements? Well users are going to use the minimum size in predictable patterns and just increment a number at the end. Those old password hashes you have to keep around to check if the user is reusing the password? Those are a liability that, when broken, tell the attacker which pattern each user is using. Not the case anymore and there is a lot more usable security rolled that is entirely transparent to end users or almost entirely transparent.
Think about how prevalent and bad captchas used to be on the website and how easy they were to circumvent. Cloudflare's and Google's captcha solution are pretty transparent and has much greater efficacy than the old ones.
Did Microsoft's general and on-going laxness contribute to bad security practices? Absolutely, but that is one ecosystem that had weird other by the nature of how inherently unstable that environment was and is not and hasn't except for maybe a brief peak ever been a core foundation of the internet infrastructure, just enterprise infrastructure unfortunately. They definitely never got the memo about usable or transparent security. I hope they're at least trying behind the scenes now.
Correct, but on evidence and in practice it's a totally different matter.
Read my other posts here, especially my comment on physical security vs IT security. Unfortunately, the evidence backs my assertions.
"Show me the incentives, and I'll show you the outcomes." - Charlie Munger.
We do not incentivize companies to operate secure, redundant, reliable computer systems. We incentivize companies to make the number at the bottom of the spreadsheet beat the expectations some analyst in Lower Manhattan set 90 days prior. And since companies handle the majority of societal work in the United States, that's how most critical systems are designed.
Now, there's a chance that this will play out in court, and that Crowdstrike will have to be bought out to make up for the damages their customers suffered starting on July 19th. However, that will take years, and the outcome could very well be that the plaintiffs will receive symbolic or even no damages. By then, the market will have hedged, captured regulatory authorities, cut its losses, and just altogether moved on. The assets will be purchased in a firesale by people who see this as "creative destruction" and won't care that peoples' lives were put at risk because of this.
And the cycle will continue.
The question is what can be done, if anything. But I've a solution in my wildest dreams as a dictator. :-)
If our culture had more respect for elders and/or thinkers that could be a start, but even then it would still be an uphill battle in a capitalist society.
Not if laws were like the Monopoly square that has 'Go directly to jail' stamped in bold all over it.
Just a few decent lockups would put the shivers down the backs of those so included.
Trouble is governments have failed to implement the necessary laws. Unfortunately, as we've seen Big Tech is too big, too powerful, and too money-rich to be challenged effectively by governments.
I agree with you the latter could be addressed through accountability, but I struggle to see what kind of law would work the way you intend here. In general, regulation helps large corporations and because they have resources to maintain nominal compliance, as well as the layers/lawyers to maintain plausible deniability if things go sideways. Regulation tends to undermine competition which further cements their power and has many negative effects that span well beyond obvious failures due to poor engineering practices.
Big Tech is now so big and powerful that it essentially does what it does with impunity, fines for breaching laws are just a part of doing business, they have negible effect on the bottom line.
The way of fixing the problem is not only to hold companies who violate the laws responsible but also equally so its employees, external advisers, accountants, etc.
Combine this with reqiring people responsible for certain corporate functions such as those who make policy decisions with respect to the way corporations police laws, check for breaches of the anti-trust/monopoly act etc. to be licensed similarly in the way electricians and plumbers are licensed. Take away their licenses and they'd not be able to carry out their Jobs.
I reckon this will eventually come to pass but I'd venture it'll come to Europe long before the US.
The issue is with ethics and being fair and giving everyone a fair go. And for that companies have to behave ethically and within the law. Right, most would say that's just being naïve as that's not how the world works in practice, and I'd agree. And that's why we have laws, they ensure some semblance of balance or order is maintained. The trouble is that in a capitalist society where competition is encouraged that 'balance' can easily be tipped. And it's dead easy for this to happen, especially so these days given there's so much money at stake. To get the edge it's more than enough encouragement for players to start acting underhandedly.
I won't pursue that further because many books have been written about it except to say I don't believe we'll ever achieve an ideal world where everyone acts reasonably and fairly. I'd also suggest that living in a completely ideal world would be intolerable, we'd lose all sense of objectivity. Society needs some degree of things not going right or not working correctly to keep it on edge.
That said, I'm of the opinion that we've gone too far in the dog-eat-dog race to the bottom and that we urgently need a correction. This can't be just left to corporations to correct through self regulation because it won't happen, and more to the point society's view of the role of corporations has changed over the last 40 - 50 years.
When I was growing up decades ago most people perceived that corporations had a dual role which was to benefit both shareholders and society. That view has shifted—or least it has in the corporate world—to that where a corporation's primary or principal raison d'être is to maximize shareholder profits. The evidence is clear, for one, it's why Boeing is in trouble—its accountants now wield the power and these days engineers have precious little say when it comes to the amount spent on safety margins, etc. The consequences of the policy shift are now becoming obvious.
It's up to society to redress this imbalance. If those in corporations have broken existing laws then the Law shouldn't ignore it (as it seems to have done with antitrust laws in recent times). Not only should violating corporations be brought to heel and punished but so too should the perpetrators who drive them (corporations don't magically do things without human direction).
Nevertheless, I think it would be counterproductive to conduct a witch hunt. Instead, we need stronger, less ambiguous laws that restate the rules very clearly. It's just not good enough to assume that most people are both reasonable and ethical because there'll always be those down at the end of the bell curve who'll always push the limits. These people must be told in no uncertain terms what rules are and of the consequences of violating them. As I see it, society (hence governments) have not done enough to ensure this happens. And I'd argue, at least in part, why it hasn't happened is because of the shift in business ethics since the 1980s (for reasons mentioned in my earlier post). Evidence suggests that business practices are now so askew and out of balance it seems we're well overdue for society to correct them.
I recall a story from about four decades ago that emphasizes what's gone wrong with corporate ethics (which was somewhat of a shock to me when I read it). First, let me say that I read this quite a while ago so I may be contorting the facts somewhat. Also, I'm now uncertain where I first read about it but I think it was either from the columnist John Dvorak or Robert Cringely in InfoWorld. (Please don't hold me to that if I'm wrong.)
It concerns the second-sourcing of Intel's 8088 CPU (back then, government required second-sourcing of components to guarantee supply), and one of the second-source suppliers was NEC. Intel and NEC entered a patenting/cross-licensing agreement in the mid 1970s so that NEC could make the 8088. This meant NEC had copies of Intel's masks for production.
NEC thought it could improve on Intel's 8088 design and without obtaining Intel's agreement it took the liberty of launching its own chips, namely the V20 and V30 which were over double the speed of Intel's offering. Needless to say Intel was rather miffed and accused NEC of copyright violation for having reverse-engineered the 8088's microcode and used it in its V-series processors.
The scuttlebutt was that either before the matter went to court or afterwards during the private out-of-court settlement negotiations the conversation between both parties went something to the effect:
Intel: "You reverse-engieered our microcode for your V20 thus violating our copyright, this was outside our cross-licensing agreement ."
NEC: "Prove it."
Intel: "Whilst you rewrote the reverse-engieered code to hide and obfuscate your tracks, there was one small bit where you didn't. As it was a bug that we'd not removed, you did not know how it worked or why it was there so you included it just to ensure things worked property. This gave you away."
NEC: "So what, so now what are you going to do about it?"
Intel: "Sue you for damages."
NEC: "OK, even before production we'd anticipated you'd likely sue us so we had to estimate the all-up costs and whether it would be economically viable to proceed. We did the sums and figured out that in the event of you taking legal action it would take you x years to obtain judgment against us in the US court system and by then we would have not only amortized our development costs, paid you reparations but also we'd have made enough profit from our V-series processors for our actions to have been well worthwhile. We just made a pragmatic decision that made economic sense."
Note: the emphases are mine.
I'm unclear whether my summary has bearing in fact or is an apocryphal account given by columnists who were reporting on the case back then, but the way I've recounted it here is what I took away from those news reports.
What's key about this account is that a large public corporation would actually stoop so low in its business practices and so act in such a dishonest and disingenuous manner but also that it was prepared to get caught and that this was deemed or considered as an acceptable or valid way of making a profit. What's even more telling is that there was no large public out cry when it became known.
My point is we have to accept that the types of people who run companies and set their policies will likely always think like this and that they'd so act if given half a chance (especially so if their bonuses are linked to profits).
That said, at present there are insufficient checks and balances to ensure these people will quickly quash any such ideas the moment they come to mind. The only way I can see this happening is for society to deem such behavior to be so unacceptable that it pushes for laws that are strong enough to both sanction corporations to the extent that shareholders will revolt and that the perpetrators will be punished with actual jail time.
Trouble is I don't see such laws being introduced anytime soon. It's possible they will eventually, but unfortunately I'd venture that won't be until after things get much worse.
And do the markets really decide? Do you really think the C-suite of Crowdstrike is going to spend the rest of their lives destitute for the losses they caused? Of course not. We have laws on the books that limit liability of businesses in these situations, and the "let the market decide" crowd are the first people to tell you these laws are a good idea because you can't possibly expect George Kurtz to do business in an environment where his 3 billion dollar fortune could be completely wiped out as the result of a court case, no matter how much damage his company did.
Meanwhile the people who were screwed by this whole thing will be lucky to get a few grand out of a class-action judgment or settlement in five years.
Markets _never_ actually decide. Not in a way that makes peaceful human society possible. You have to introduce systems to give minor players a way to redress grievances, or they'll find their own, often through less-than-sporting means.
(1)
Citation needed
Except in the gambling industry. As part of a long-standing tradition, companies in the gambling industry are usually contractually required to take financial responsibility for errors. GTECH's annual report, before they were acquired by an Italian company, says "We paid or incurred liquidated damages with respect to our contracts in an amount equal to 0.61%, 0.18%, 0.50%, 0.47% and 0.14% of our annual revenues in fiscal 2006, 2005, 2004, 2003 and 2002, respectively."[1]
So, forcing a transaction process service to take full responsibility for errors cost, at worst, 0.61% of revenue. This is sufficient to force gambling companies to use unusually good security technologies.
The Nevada Gambling Commission has technical rules.[2]
* "On-line slot systems may only communicate with equipment or programs external to the system through a secure interface. This interface will specifically not allow any external connection to directly access the alterable data of the system." Which means no privileged "security" systems such as Crowdstrike.
* "Gaming device application access to the system based game must be logged automatically on the system component of the game and on a computer or other logging device that resides outside the secure area and is not accessible to the individual(s) accessing the secure area." Which means the really important info must not only be logged, the logs have to be kept where the people who run the systems can't get at them. There are more logging requirements. Most things require two logs, one used for normal operation and a remote backup with tamper resistance and secure hashes.
* "Conditions for changing active software on a conventional gaming device or client station that is part of a system supported or system based game: (a) Be in the idle mode with no errors or tilts, no play and no credits on the machine for at least two (2) minutes; (b) Not be participating in an in-house or inter-casino linked payoff schedule..." There's more, but the general idea is that to change anything, you have to take the component being changed down to the idle, fully backed up state. Only then can changes be applied. All of which are logged.
The gaming industry has faced hostile actors for decades. They have reasonably strong defenses. Yet they're still very profitable.
[1] https://www.sec.gov/Archives/edgar/data/857323/0000950123060...
[2] https://gaming.nv.gov/uploadedFiles/gamingnvgov/content/Home...
The 90s and into the early 2000s at least. You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet.
Perhaps this happened where you were, and lucky you it seems you were in a good environment.
But back then I was in IT management and I had precious little power to stop it especially given other senior managers were the culprits. The operation had another function and not IT as its primary role. Moreover, I saw very simular problems in other organizations that I was familiar with.
Also, during that period I was with another outfit whose principal function was surveillance—not of people but of info and physical stuff and I can assure you that whilst the system worked well try as we might it wasn't watertight.
No offense, but I think you’re thinking of a later era. Most protocols back then had literally no auth or auth that was never deployed. The thought of critical systems safe enough to be exposed to the internet was just really unfathomable.
I miss the days when any application that dared to phone home even just to check for updates was considered spyware. Today there is are huge numbers of people who have access to install and run whatever new code they want on our systems whenever they feel like it. If it's not the AV software, it's the browser, or the video card, or the mouse driver, or windows itself. It's totally unmanageable.
> Was there ever such a time? If so then tell me when it was
It seems very plausible that "digital resilience" that this has been buzz phrase repeated often enough in meetings of security-adjacent corporate bureaucrats that some number of people convinced themselves it was a real thing.
And the same divorced-from-specifics approach allows these decision makers to paper over any and all choices that inherently weakened security 'cause the triage needed to partially protect the resulting structurally insecure system can be presented with similar glowing buzz phrases.
In case you don't know, Crowdstrike is hardly the only company with large scale access to this many companies,governments and resources. It takes one rogue employee to deploy a disk wiper that destroys every computer (including linux and macos) and affected systems won't recover at all. it would be months before critical systems are back online, the global economy would come to a halt worse than how it did with COVID in such a scenario.
It isn't "why didn't Crowdstrike do better" (although they should have), it is more, why isn't technology in critical systems more resilient to one vendor screwing up or getting hacked?
For example, let's say it wasn't just a boot loop but a disk wiper erased every boot disk, is there any reason pxe booting a recovery image or a backup image configured already on servers, atms, kiosks, point of sale systems,etc...? even if UEFI and bios were erased, it is technically not impossible to have an auto-recovery mechanism implemented right?
If you have never been in an incident response (IT and security incidents) root cause analysis, I don't blame you for not thinking deeper about the root cause, but that is the type of root cause analysis that has been missing despite over a decade of rampant ransomware, disk wipers, and supply chain risks.
Finding someone to blame and be angry at is easy and doesn't solve the root cause. Making hard technical decisions and not wasting this opportunity (never waste a good crisis) to push for resilient technology investments actually solves the root cause behind this and other repeating problems.
imma take your comment one step further and say that the emphasis on security is coming at the expense of discussions on resilience. and security matters a lot less, especially financially, than resilience.
Backup firmware and boot images can be configured as read only.
Everyone in the industry knows this.
Interesting to see the NYT just catching up.
Maybe it has to do with some major incident that happened yesterday, and the fact they are a news company?
https://www.nytimes.com/2021/02/06/technology/cyber-hackers-... https://www.nytimes.com/2018/01/03/business/computer-flaws.h... https://www.nytimes.com/2013/07/14/world/europe/nations-buyi... etc
NYT: Boeing is run by bean-counters and isn't taking engineering seriously anymore.
Boeing: That's not true. Our aircraft fly thousands of times a day, every day, and are very safe.
Who would find that very interesting, absent any relevant, dramatic current events?
Everyone bitches about regulation and taxes, for reasons real and imagined, but applying laws and rules to businesses before something happens is the point of them.
Isn’t it more of a problem of the population at large?
This bears repeating.
If you complain about a risk before a disaster structs, you're fearmongerng.
If you complain about a risk after disaster structs, you're flogging a dead horse.
In order to write about Boeing they'd have to have an angle and resources to go on a fishing hunt to create an interesting story for people to read and talk about.
[1] https://en.wikipedia.org/wiki/United_States_Foreign_Intellig...
The lengths that the NSA and CIA would go to to implant backdoors (interdicting shipments of laptops/phones and doing the work themselves) further suggests that they cannot compel this sort of action.
If they have a path to covertly compel action as a state secret under National Security / anti-terror laws we will only hear about it from whistleblowers. It won't be something the target can disclose let alone test in court.
FWIW I also don't believe in Apple's nobility as resisting on user's behalf. They happily bow to the state and remove apps for e.g. organising protests, monitoring deaths in US wars, csam scanning etc. IMHO their interest in encryption is to prevent jail-breaking and protect their app-store cash cow.
> interdicting shipments of laptops/phones and doing the work themselves
I don't think that proves anything about their powers. Given the option, I'm sure they would prefer to install things themselves without third-party knowledge or consent.
We have evidence of complicit action e.g. black rooms like Room 641A. I think the nature of "consent" and "obligation" gets pretty grey when it comes to the security agencies. They don't get results using court orders. I'm sure they have assets employed as staff in security sensitive positions.
https://cbr.ru/eng/press/event/?id=18776
https://bank.gov.ua/en/news/all/drugiy-rik-povnomasshtabnoyi...
At this point it's not cash that is missing, it's absolutely everything that sustain human life, and there's probably not enough working things to even barter.
In comparison, total KIA losses of Soviets in the Afghanistan war were 14k-26k, and Americans in the Vietnam war lost 58k KIA + 150k WIA throughout 10 years.
In short, this is the biggest war in Europe since WW2. But hey, it's not war enough because not enough Ukrainians are dead or something, idk.
[~] https://t.me/pechalbeda200
[^] https://meduza.io/en/feature/2024/07/05/a-new-estimate-from-...
One of the worlds (what we thought) super powers has been trying for the last two years to destroy the infrastructure of a country with 33M inhabitants. They may not have fully mobilised, but they are definitely spending all their military equipment. Long range / tactical missiles. Air assets. Naval assets. Cyber warfare.
The result in Ukraine : unimaginable human suffering, but electricity and the internet are still working over there.
When the nukes start flying, that's another matter though. But in that case our problem will not be that our credit cards stop working.
But in terms of total assets deployed, they are currently all-in. Attrition being what it is, they are reducing their Soviet stockpiles at a prodigious rate and are currently activating 40’s and 50’s equipment.
This is all extremely well documented by open sources. People are counting tanks on military bases using satellite images. Check out Perun on YouTube. He’s a defense economics expert that posts a 70 minute PowerPoint presentation every Sunday, complete with sources and references.
The high-speed "Sapsan" trains from Moscow to St. Petersburg are German, not the other way around. The cranes they used to build the Crimea bridge were Dutch, not the other way around. The optics they were putting in their newest tanks was French, not the other way around. But of course, all these guys need to take over Europe is just "mobilization", whatever that means. With multiple times less people and inferior technology. Right.
You are either seriously misguided or just trolling.
I would guess that NATO planners have been adjusting their assessment of what Russia is capable of when completely mobilized. The answer sure looks a lot like "way weaker than we imagined, pretty well ineffective in the face of significant resistance, the only reason to pay any attention them at all is they have ~half the nuclear weapons worldwide."
You are claiming this is actually beneficial for the Russian Federation because all of that is outweighed by experiments in tactics, correct?
[_] https://www.moscowtimes.ru/2024/07/15/mvd-otchitalos-oroste-...
While we focus on democratic debates that are based on their spoonfed misinformation campaigns, Wagner is literally conquering central african countries one by one.
Russian production numbers sound good if you ignore that most of them refer to stored Cold War era equipment being reactivated. Their main and most successful staple has been artillery and that mostly worked because it could be fired from the safety of being on a side of the border NATO pretty much told Ukraine not to cross - for a time. It also seems like the "saber rattling" Russia did in the lead-up to the invasion by positioning military around the Ukranian border was less of an intimidation tactic and more of a necessary part of the process.
I'm not saying Russia couldn't do a lot more damage in an all-out war into the West even without involving nuclear weapons (which already assumes European countries with nuclear weapons or the US wouldn't use them either). But based on the underwhelming performance of the Russian military relative to its supposed numbers, I don't think Russia could have pulled off the kind of Blitzkrieg you're envisioning, let alone once supply lines become a problem. Especially if you consider that the plan for the invasion of Ukraine clearly was built around a surprise attack on Kyiv, which failed spectacularly because the terrain and weather meant the tanks had to drive slowly in a line and somehow Russia didn't bother providing infantry support.
OP: Looking at two countries in an actual long running war...
You: This isn’t really all hell breaking loose actual war.
Nobody used the term "all hell breaking loose". You did. You redefined the conversation to be about 100% mobilization (from Russia), and then got all pissy that people called you out.
Could Russia mobilize more? Yes, absolutely. But based on what we've been seeing they lack the supply lines, resources and frankly military competence (which is unsurprising given how Putin deals with anyone but yes-men) to be able to do anything with that if they had it. Also, as I said, the NATO response to invading, say, Poland would also look extremely different from the current NATO response to invading a non-NATO border state.
The Soviet Union's main advantage during WW2, other than receiving immense support from the Allies (both in supplies and military equipment), was that it was fighting the Nazis on their back foot. The Nazis had made a similar fumble as Russia did in Ukraine by misjudging the climate and seasonal weather and they also didn't have a reliable supply line. The Soviet Union did deal a devastating blow in Stalingrad but the Nazis there were pretty much stranded in hostile territory without support at that point and many soldiers were suffering from frost-related health issues. When the Soviet Union actually invaded Germany the Nazis' troops were already spread out all over Europe and into Africa and losing ground in the various occupied territories.
The Soviet Union suffered massive casualties and contributed more sacrifices to defeating fascism than any other country involved, but militarily its capabilities do not translate into any scenario involving modern Russia NATO analysts would lose sleep over. Russia is "holding back", yes, but so is NATO and especially the US. Even the "modern" equipment the US has started sending to Ukraine is decades behind on what the US military has available. Ukraine's stronger allies have very much been holding out on "the good stuff" and instead mostly cleaned out their dusty reserves. Russia OTOH doesn't seem to have the production capacity nor resources to churn out its newer equipment (which is still years behind what the US etc have access to) and is already falling back on decades old stock and desperately buying ammunition from North Korea of all places.
No lack hostile hackers.
Outages are largely limited to physical infrastructure that’s attacked by missiles. Russia isn’t a slouch in digital warfare, either.
But the point is valid.
If more of the critical machines were running different OS's, the damage would be contained.
When we talk about the dangers of "monoculture" it's usually about plants. The same danger applies to computing infrastructure.
Sorry for being harsh with my following statement, but I believe that the companies affected by Crowdstrike share some responsibility on what happened yesterday.
The threat is inside the building!
And if it hit your system, the vendor's first response would be "are you on the latest update? that's been fixed."
(Unless the latest update IS the problem. In that case, being lazy was a good defense.)
- Whats bad is instant global 0->1 rollout, instead of more gradual, blue/green/canary however you call it. With gradual rollout policy this whole thing could have been caught at their first couple guinea pig customers, and not the whole world
I'm sure IBM mainframes are still running critical stuff, too.
Not if they were running the same CrowdStrike.
"regardless" ?
So you don't actually know, is what you're saying.
Windows is shit.
Mac is more or less.
Linux is best of all.
"Network effects"? You mean like, "I'd be fine, but I depend on a service from a Windows machine, so I'm still screwed" ?
> The number of platforms you'd need for this much safety is impractically high
I don't see why this becomes an impossible problem. If all the essential services are not provided by a single software infrastructure, then we have the required diversity, right?
In the case of airports, losing ATC at just a few major US airports would effectively paralyze the network. Or yes, the case you mentioned where you depend on four SaaS offerings, and odds are one of them will go down.
It's ironic that the original DARPA justification for packet-switching was, if a nuclear war takes down some nodes, the packets will still get through somehow.
I wonder if this will teach absolutely anyone a lesson about anything.
The response is and always will be - how much will this cost. We now have the opposing figure, how much will this cost if we don't do it.
> “We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum.
I disagree. If the only hope is some vague promise of bs AI, there is no hope indeed.
I don't think AI would have done a better job convincing CrowdStrike's leadership team that a staged rollout strategy is worth the cost.
These issues are caused by people, not technology, so more technology isn't going to fix it.
The purpose of a system is what it actually do, not what it claims to do but fails every time at that. Turning everything to vulnerable as fragile with some big strategic and global plan ahead makes you into a disposable asset, a sacrificial victim in some higher level chess game. And you can agree with that with your decisions.
I don't have an answer, but thinking about it makes one understand how incredible fragile our complex logistic chains (and indeed our economy) are. One day all this complexity will collapse upon itself and we'll wonder what happened.
I don't think this is linear though. It's easy to target a weak point to inflict a small amount of damage, but hitting say 10% of GDP would mean targeting multiple sectors of the economy and putting millions of people out of productive work.
The ability to get the bomb in the right place is far more important than the quantity of explosives, as was demonstrated by the recent suicide sniper missing.
The IRA https://en.wikipedia.org/wiki/1993_Bishopsgate_bombing was estimated to cause more economic damage than all other IRA bombing put together. It's interesting that (apart from the first WTC bombing) American terrorists have stuck strictly to guns and not attempted car bombs.
That is not true. Oklahoma City bombing is the first which comes to mind where the explosives was planted on a truck. But there are many others, there is a whole wikipedia list about them: https://en.m.wikipedia.org/wiki/Category:Car_and_truck_bombi...
16 is not "many", and quite a lot of those are decades old. Whereas https://en.wikipedia.org/wiki/List_of_school_shootings_in_th... is just an endless scroll. It's a form of terrorism that America accepts as routine as car accidents.
I agree terrorist in the USA use guns a lot more than vehicle-born explosives. If that is what you would have written i would have not commented anything.
But I hope to see it in my lifetime.
Already it's amazing how the media is presenting this like it's a natural disaster, instead of an entirely preventable display of incompetence... A business entity whose shares only dropped 10% after causing untold billions of damage to the economy.
Gives us all a bad name.
Amazing? Super predictable I'd say.
> Gives us all a bad name.
That is sadly true.
Yeah agreed. It would require no corruption... which is the true fantasy trope of our times.
Systems like CrowdStrike are mandated, not hired freely.
He believes that just like in the medical field, the software industry must self-organise before government start imposing draconian measures about how software should be developed.
2. Also a great time to start prepping for AI Incidents => https://thedataexchange.media/ai-incident-response/
If general IT had the abilities of sales, marketing, or insurance, there might be a chance that the business would take the responsibility to have the internal knowledge and capabilities to assert control over their systems. But they don't, and as such they won't and instead shove that responsibility over to a third party generalist elsewhere with enough paperwork to have both parties feel their asses are covered.
As long as everything seems to be working, the signals that are still getting through is project failures, be it complete failures or just time and/or money being consumed more than planned and maybe some requirements getting cut. But as soon as enough stuff breaks at the same time, we get news outlets writing articles about resilience and the greater public suddenly no longer agreeing with that is effectively just the result of the status quo because it impacts them directly.
What is the response of a Free Press to news stories exercising reflexive blame-game from allied core groups with major monetary interests in the outcomes?
Yes, it's illustrative of the USA. Due to monopolies, lack of local control of infrastructure etc., a feature is rolled out that grinds hospitals, airports etc. to a halt. Surely due to forces we're surely familiar with - a rush to get profit-making features out, a neglect of correctness and stability, cost cutting etc.
Then we have the New York Times, considered the sober voice of the establishment. What is discussed? Reflection on how entirely US-internal corporate processes led to this? No. A thought experiment about what if some external actor, perhaps one tired of US imperialism or something, had performed this.
I read this after seeing Hulk Hogan rip his shirt off at the RNC in an Idiocracy prophecy manifested, while the other presidential candidate immersed in the dementia of the gerentocracy clings to power amidst his cohorts pleading he step aside.
As I watch the US arming the Ukraine to fight Russia, I think back to 1986 and Gore Vidal's plea for an alliance with Russia lest Americans become either farmers or just entertainment for the more efficient Asians. Another prophecy which seems due to come to pass.
As much as some people want to believe that Microsoft is blameless here, I hold them partly responsible. They need to create a stable API in their kernel and force third party security vendors to use it.
This situation was caused basically by an anti virus definition update.
Without any information, my guess is that CrowdStrike specifically doesn't provide a means for enterprises to manage the CrowdStrike updates because that would cause potentially weeks or months of delays for critical anti virus updates to be released.
I'm more curious why they need CrowdStrike on every system in the first place. I can understand employee computers but servers and critical systems should have other security measures in place to make them less available to attack in the first place.
The boring answer(s) is compliance and necessity. You need a security solution installed & running for a bunch of certifications, and maybe you also need it to secure some unmaintained software you have to run. Plus, even if the machine isn't front-facing, a lot of these EDR solutions are installed to prevent or monitor for lateral movement.
I have a different take. This was still far from being an adversarial attack. There was no security breach. The failed configuration came from an SDLC that remained secure and fully in control of CrowdStrike. It was a terrible bug, but it was not an attack
There was no validation, phased roll-outs, almost certain no multi-person verification. I'd bet dollars to donuts this was pushed out by a low/mid-level functionary that could be carried out by dozens if not hundreds of employees. There may have not been a security breach, but it was still one minor security breach, distracted open laptop in a cafe, or disgruntled/paid-off inside actor away from absolute armageddon.
It wasn't an attack, but it was a raccoon who came in through an unlocked screen door in the back of Fort Knox.
If someone had used this to deliver a ransomeware package, they'd be buying a mega-yacht right now.
EDIT: It is in the 'drivers' directory, has a .SYS extension, but was something called a "channel file" but I couldn't get much info on what a channel file does other than "something something named pipes"
Web + chrome is so much better. Then just use qnx or something for embedded. Why is the actual reason that our $600k confocal microscope has to run windows?
Qnx or Unix is much better for scientific and healthcare equipment.
Yeah, like eBPF in Linux or System Extensions in macOS.
“We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum.
On the other hand - it's d*mn hard to imagine that any of America's "A List" or "B List" adversaries didn't have a far-more-detailed road map, years ago.
This is actually a really useful hypothetical standpoint to work out security from.
Designing systems that start from the assumption of insecurity helps us build more robust protocols and management. Qubes OS starts from the position that all VMs are or soon will be compromised. Zero-trust in network design assumes the bad guys already have the whole network. Plenty out there would like to shrug and say "the endpoints are all rotten too" (especially with phones which are a veritable hell to secure) and move trust into the application via trusted execution methods.
> and nothing can be done about it?
No, That doesn't follow. It's prudent to be realistic about threats. but there's always a way out, at a cost. The cost, in a complexity crisis, is throwing away a lot of what we've done.
But it seems that reducing costs is more important even compared to preventing people on life support in hospitals from dying.
What a world.
So what does Qubes OS do to protect against a hypervisor bug? Those must exist.
How do you ensure that your systems are still working and retrieving data from databases, etc, when bad guys have the whole network and can block all communications?
"CrowdStrike debacle provides road map of American vulnerabilities to adversaries"
My assertion: America's serious-threat adversaries already had far more detailed road maps, years ago. The intel value of whatever "road map" data they got from the CrowdStrike debacle was pretty marginal.
Neither the HN Title nor I said anything about backdoors. And within 2 para's, the NYT story makes it clear that CloudStrike's Big Oopsied had nothing to do with bad guys hacking anything.
Turns out our homegrown CrowdStrike was just as bad as our fears over Kaspersky were. Perhaps worse.
Citation? I thought it was the "other fears"; this is the first time I'm hearing accidental outages were one of the concerns.
Naturally, the question of malicious intent would most likely be more or less prevalent depending on whether country X is considered an adversary.
Cute. It's always those bad keystrokes. If only these crowdstrike employees worked on their good keystrokes that morning. I blame management.
> Russian hackers working on behalf of Vladimir V. Putin bring down hospital systems across the United States. In others, China’s military hackers trigger chaos, shutting down water systems and electric grids to distract Americans from an invasion of Taiwan. ... Among Washington’s cyberwarriors, the first reaction on Friday morning was relief that this wasn’t a nation-state attack. For two years now, the White House, the Pentagon and the nation’s cyberdefenders have been trying to come to terms with “Volt Typhoon,” a particularly elusive form of malware that China has put into American critical infrastructure.
So we have cyberwarriors and cyberdefenders? And the russians, china, etc have 'hackers'. If ever there was a doubt what the nytimes really is.
> The fear is, in an election year, that the next digital meltdown may have a deeper political purpose.
Oh dear. More bad keystrokes on the way?
Did anyone glean anything of value from the article? There was a lot of words but no substance.
When will we learn?
You hate Rust -- fine (not fine but OK, I guess people get super triggered over it and it's a reality I can't change but I am still baffled by it because they throw away reason for emotions and these people should really know better). Fine. Just use Golang or any other GC language really (Java or C# as well, if you must).
When will we abandon convenient routine and start adapting to modern realities? ("Modern" being at least 25-year old here but hey, I am willing to give you some leeway and not roast you too much. Let's assume these are "modern" realities, f.ex. just the last 5 years.)
Not saying you can't write bugs in Rust, of course -- that would be crazily delusional. I am saying they needed a better process. And I am saying that a stricter language could have improved the process a bit as well.
Doing the same with zero control over the non-Rust side of the kernel seems completely untenable.
Being a multi-million dollar company and using unsafe languages today is not a good look. But everyone gets away with it because everyone else is doing it.
The root issue is giving privileged access to a business entity you think you can trust, but clearly can't.
I'm a fulltime Rust developer, but I don't think Rust saves you here.
I mean that the least you can do before pointer dereference is just check for several bad sentinel values, NULL being one of them.
Seems like a rather amateur mistake to me.
I've been an idiot as well in the past. Happily some of us actually learn though!
E.g. loading it would require you to setup a maximal size and a valid configuration struct?
We haven't seen the code but it could be something like:
char *ptr = parsefile(file_we_released_without_testing);
if(ptr[0]=='A') { } // BSOD loop
So this style of error can be addressed by using a safe language. Or static analysis. Or code reviews. Or not doing this stuff in the kernel. Or formal methods. Or fuzzing.
As someone else said you likely can't easily use Rust for Windows kernel modules/drivers. I'm sure a strong enough engineering team could do it (e.g. transpile Rust to C) but I'm not sure it's the biggest engineering problem CrowdStrike has. Microsoft has a complete tool-chain for developing these and it's usually C/C++ or assembly.
func parsefile(string) string {
}
func thatfunctionthatcrashedinC() {
defer func() {
if err := recover(); err != nil {
log.Println("panic occurred:", err)
}
}()
result := parsefile(badfilethatcrashesC);
if result[0] == 'A' {
}
}
Ok, but seriously I don't believe this will ever happen and I don't really think this is a language debate nor do I want to engage in one.
This is about putting critical infrastructure connected to the internet that's running an operating system that you can't trust out of the box. Since the Windows OS is susceptible to so much malware you need all these third party services (which you also can't trust or audit, but it's absolutely better than not having anything) on top of the OS.
There was a whole host of companies that had zero problems, not because they're using Rust, but because they have much better security practices and quality infosec employees.
Agreed, they should not be using Windows in the first place. That should have been the first line of defense.
> There was a whole host of companies that had zero problems, not because they're using Rust, but because they have much better security practices and quality infosec employees.
Fair enough, I only commented on one layer of the security stack -- so your remark that expands the scope is valid and welcome.
> We're all waiting for your anti-malware Rust Win32 kernel module...
I am done working for free. If I am paid to do it I am sure I would have done better than this poor confused soul who allows NULL pointer dereferencing which is a mistake that most C/C++ interns quickly learn to avoid.
Crowdstrike borked RHEL 1 month ago https://access.redhat.com/solutions/7068083 Literally the same situation, unbootable machines.
The reality is that shitty software broke everything. Why do we have to drag the OS into this?
But I am in full agreement with you that sloppy programmers cannot truly be helped. They just screw up and move on like nothing happened. Sigh.
Something of this nature would have had our entire team fired. The number of phases and the thoroughness and exhaustiveness of the protocols we have to ensure we don't push bad builds would have most engineers taken aback... but we have to. With great power comes great responsibility.
I have to imagine that this bug has existed for quite some time and I’d be curious to know what other input validation errors they have, considering the amount of untrusted input they evaluate at ring 0 originating from userland.
At the very least, big money security software companies should be parsing untrusted content with some kind of rigorouly safe approach, not just squirting it through a big pile of C/C++.
And don't get me started on the whole concept of undefined behavior in those languages. To quote I. I. Rabi, "Who ordered that?"
the malformed files were updates from crowdstrike itself. It's not exactly "untrusted content".
[1] https://devblogs.microsoft.com/oldnewthing/20220907-00/?p=10...
I'd love to hear from an engineer on the project but unfortunately we're likely not to.
BTW, using your metaphor, until 2 days ago they didn't even know that there was a fire in the basement, nor a basement.
To screw up so legendarily requires a concert of bad decisions.
C++ was a terrible, terrible mistake.
It's good to criticize the current state of things, but don't pretend you have a solution.
Also, do we know if rust would have helped here? Rust doesn't guarantee no crashes -- in fact, panicking (aka crashing) is the default.
RE: panic default, don't get fooled by hobby projects, professional Rust code always does pattern matching and does not defer to panics.
The "software that doesn't exist" point is somewhat valid, though it's also the chicken and the egg problem, as in that not many people are working to make it happen because the current state of affairs is wrongly deemed as good enough. And it really is not.
That's a "programmers who know what they're doing don't make that mistake" argument. If that were tenable there'd be no need for rust in the first place.
But this is easier to scan for compared to all the potential memory unsafeties in C/C++. It's an improvement.
2. If the driver crashed due to a Rust panic, the result (boot loop) would be the same.