CrowdStrike will be liable for damages in France, based on the OVH precedent
thehftguy.com
thehftguy.com
OVH was held liable because of the data loss, not for the service interruption. Data loss is something irremediable, permanent, definitive. Some businesses were basically ruined from this incident because they had no more data to operate. To add insult to injury, they sold offsite backups in the datacenter literally meters away. A service interruption, well, shit happens, and this is handled by SLA contracts that both parties agree to. You don't ruin a business (read: close a company) for a few days of outage.
I doubt CrowdStrike will be held liable for much; from corporations at least. They cannot repay the damage done, or they close the door. The healthcare sector is another beast, but I think it will come to more regulations for critical entities.
like if the largest outage in history was caused by you due to a config parser failing and it looks as far as I can tell that they didn't follow industry best practices when it comes to config/parsing handling and probably also didn't follow some best practices when it comes to kernel module programming then honestly it would be really strange if you didn't had to declare bankruptcy due to damage payments (which doesn't mean the software is now gone/unmaintained, there are a lot of ways to make sure that doesn't happen, e.g. MS anyway had interest in buying Falcon).
I thought the same until I saw the damage estimates. They’re in the single-digit billions. That’s well below CrowdStrike’s market cap. Unless we’re going hard for retributive justice, liability should be enough.
(through yo have to make sure you QA team works properly, i.e. in tandem with your dev not in fight with them, to many QA teams are a mess, but QA is important)
But this only creates excuses for all other responsible players in this systemic issue - or society at large. Just to pick 1 example: I keep reading comments on how profoundly health care providers were affected and that lead to human life losses.
I understand that having "tech" involved in health or any sector is important but are we really wanting to build critical services that grind to halt or have huge efficiency impact when a single vendor fails? Are these service providers not responsible for thinking about failure modes?
[citation needed]
If aws goes down in a region yeah it sucks but we fail over. If aws goes down ww then it's like well...sometimes that happens. If I've built critical infra like electric utilities, airports etc with swathing vulnerable points then that's the real problem.
Logic dictates that the more critical the data - the more likely it is gone. An informal anecdotal survey says that alot of users use bitlocker - which means a lot of data loss.
EDIT: I see that in many cases one can recover Bitlocker encrypted drives. I wonder how much real data loss there is.
The root of issue, not understanding local laws/culture, is very similar - surrounded by a vast market/culture (US +Canada) dulls your senses for the rest of the globe.
US companies don't think about Canada as anything but an afterthought, and struggle with the same issues here that you just mentioned.
Canada is metric, uses different spellings (closer to UK English.. colour, not color [Chrome just marked my spelling as wrong despite me having Canadian English as my setting]) and is officially bilingual with localization laws requiring companies to provide French versions for certain kinds of services. US companies either don't bother, or usually get this entirely wrong.
e.g. It's been how many years? And navigation on Android / Google Maps can't pronounce French names for streets/places while driving around in bilingual places in Canada. Just completely butchers them, their system can't "understand" the concept that you could have your language set to English but still need to hear French place names, or vice versa.
Honestly, I think this is the right approach, and I'm speaking as a bilingual French/English speaker.
Google Maps doesn't know that you are bilingual. So it has two choices: pronounce words the "right" (i.e., native) way, or pronounce them the "English" way. If someone who is unused to hearing the native pronunciation, their understanding is going to be impeded. Google Maps tells you for example to turn onto Passage du Grand Cerf in a French accent, a non-Francophone is going to have a harder time finding that.
(Now, okay, maybe you should have a way to say that you're bilingual and want to hear the French pronunciation, but I understand why they don't do it.)
On the old lines, they had what sounded like a native English speaker do the announcements of the upcoming station, including the station name. On the newer ones it sounds like they just recorded the "we're arriving at" bit once and then spliced in the read-by-a-Chinese station name, which is much harder to understand as a foreigner. So on the new lines I was constantly staring at the displays.
Even, like, names ... Duplessis or Cartier or whatever... just completely unrecognizable.
Or take the way it reads out the exits off the 401 in Toronto, reading the full bilingual sign (where there is one) with the English first and then the French part as if it was a continuation ... in English ("Sud" read out like you'd say that if it was an English word, hard D and everything.) Clearly, they should just drop the French part, but their system can't comprehend that a sign could be bilingual. Because America.
Look, I don't speak French, I'm not bilingual, and it drives me nuts when I'm in QC or even just Ottawa etc. I literally can't tell what street it's talking about, it doesn't correspond with the sign. It's incoherent.
What's mind boggling is that Google navigation is completely fine reading out Spanish words when I drive around California. C'mon.
But it does know, e.g. "Accept-Language" header. Of course, google resent that part and travelling across Europe results in having a different language every day.
I'm not bilingual but I'm British and the most irritating thing driving through France is hearing Google/Siri butcher French words. It makes it basically impossible to navigate!
I imagine most Brits would be able to hear a French word (spoken slowly enough) and recognise it written down.
> If someone who is unused to hearing the native pronunciation, their understanding is going to be impeded
Circular reasoning. If common software used the right pronounciation more people would be used to it.
Due to the proximity of US (tooling/documentation), lots of the trades still operate in non-metric (or some unholy combo). Most folks would be a lot more familiar of PSI when it comes to tyre pressure (compared to bars), etc. I don't know if L/100km is the standard unit to measure fuel consumption (efficiency).
It's just the reality of "sleeping with the elephant" as the expression goes here. We just have it worse than Europe.
That's Fronkensteen.
It’s just a difference in travel and seniority. If you aren’t talking across continents there is no need to speak two languages.
Continents have nothing to do with this. If you live in the UK and need to talk to people in the USA and Australia, you can be monolingual and still speak with people in three continents. If you live in Switzerland, you may need to speak 3 languages just to be able to talk with all your neighbors.
My going to a conference in India and arguing over the lakh/crore system isn’t useful to anyone [1].
And in Europe there are numerous differences between countries of this kind - Germans and a few others use different number separators (1,000 is 1000 in France or the UK or Spain, but 1 in Germany or Romania). Several places drive on opposite sides of the road. The UK uses many imperial units. I'm sure there are others I haven't even come across yet.
Got it, you’re parsing continents literally. I was speaking colloquially. Read it as “cultures” in the first comment.
No, France and Spain follow the same standard as Germany: dots as thousands separators, a comma as the decimal separator. Actually most of Europe does the same, the only exceptions being the UK and Ireland:
https://en.wikipedia.org/wiki/Decimal_separator#/media/File:...
There are also a great many families in the US whose first-generation members have limited English.
This also goes notwithstanding the indigenous peoples, whose Diné bizaad and Tsalagi, for example, are also spoken here.
Am curious: where does that not happen in the US?
(And in parts of Canada they say 4-20-10 to mean 90 :-)
It happens with dates and makes them unsortable.
To us French this is a word like others, it's not like we are calculating in our heads. Belgians have "septante" which is more logical but they do not calculate either.
The worse offender on this I suspect it's Apple. Where half of their localization stuff doesn't work or works in a weird way
I have to work with logs from computers and applications and this is driving me nuts. "What is the timezone ?" is the question everyone fears.
London is in the global west, except for the whole thing with the Greenwich Meridian going through… Greenwich, east London. Not to be confused with East London, which is in South Africa, which fortunately is also in south Africa.
The UK is definitely in the west of Europe though.
What's wrong with using a calendar date like may the 1st? I know that there are other calendars too. But is more manageable IMO.
Usually it's because they want to keep it vague because the exact date (or even month) hasn't been set yet.
- use "second half of ", "begin of", 3 quartal of, etc.
- or a specific month if they want to be more precise
also for western focused announcements they also use "holliday session" as their tends to be a holliday session in most countries in both summer and winter (through their start differs _a lot_, but it tends to just work out if you release early enough)
And they sometimes use their internal fiscal year, which doesn't align with the calendar year. So sometimes, when they speak of the "fourth quarter" of an year they are talking about the beginning of the next year, or in the opposite direction, they might speak of the "first quarter" of an year but they're talking about the end of the preceding year.
That is winter in the southern hemisphere.
I'm amazed at the need to have to explain this to a grown adult: https://spaceplace.nasa.gov/seasons/en/
Classic example!
In the US, people will normally assume you are talking about Christmas holidays. In Blighty, people will assume summer holidays.
I think this one is understandable.
Americans have a very strange and tunnel-vision world view.
I blame most of this on the fact that when you turn on any news channel, in an average 1 hour new program, less than 10% of the time is spent talking about anything outside of the US (and when they do talk about international news, it's always tied back to how it impacts the US - no one is listening to international news just for the sake of knowing what's happening in the world)
This is true for every large culture.
https://europa.eu/youreurope/citizens/consumers/shopping/gua...
But after 1 year you have to show that it's the vendors fault instead of the vendor having to show it was your fault so often "de-facto" 1 year.
Various exceptions include for stuff like food, underware etc.
I wonder how this kind of thing is organised, since there's all these jurisdictions.
In theory simple. Crowdstrike is doing buisness in state X, so compensation claims will be settled in court in state X. So lots of courts and lawers all around the world, will be quite busy for some time with the case.
If that doesn't work, judgements can be registered in other courts for collection purposes.
OVH is different in that it's actually a French company.
In the end probably only the us companies will be left empty-handed.
Many businesses will simply refuse to buy your product if the contract says the dispute has to be settled under foreign law or by a foreign court. The customer's lawyers will flag such a term as an unacceptable legal risk. And if your competitor isn't demanding that term, you are giving them a big reason to choose the competitor instead.
Random example: Oracle's standard contracts with their Australian customers says disputes will be settled under Australian law (New South Wales state law) in an Australian court (in Sydney). [0] And Oracle's standard agreements for France nominate French law and the courts of Paris. [1]
If Oracle can't get away with forcing foreign law/courts on their customers, I'd be surprised if CrowdStrike can.
Might be a different story for smaller countries, but most businesses in major economies are used to vendors offering contracts under their own national law.
[0] for example https://www.oracle.com/us/corporate/contracts/cloud-csa-v012... – see clause 14 on page 7
[1] for example https://www.oracle.com/assets/cloud-csa-v012418-fr-eng-44198... – see clause 14 on page 6
this is not true
just because you write into your contract that something will be settled in a specific jurisdiction doesn't mean it's legally actually the case
In the OVH case, their backup system (as a whole) failed. Many customers were left with 0 data, and per the article "the court ruled the OVH backup service was not operated to a reasonable standard and failed at its purpose".
Meanwhile CrowdStrike "just" crashed their customer's kernels, for a duration of about 1 hour (during which they were 100% safe from cyber attacks!). Any remaining delays getting systems back online were (in my view) due to customers not having good enough disaster recovery plans. There's certainly grounds to argue that CrowdStrike's software was "not to a reasonable standard", but the first-order impacts (a software crash) are of a very different magnitude to permanently losing all data in a literal ball of fire (as in the OVH case).
Software crashes all the time. For better or for worse, we treat software bugs as an inevitability in most industries (there are exceptions, of course). While software bugs are the "fault" of the software vendor, the job of mitigating the impacts thereof lies with the people deploying it. The only thing that makes the CrowdStrike case newsworthy, compared to all the other software crashes that happen on a daily basis, is that CrowdStrike's many customers had inserted their software into many critical pathways.
CrowdStrike sells a playing card, and customers collectively built a house with them.
(P.S. Don't treat this as a defense of CrowdStrike. I think their software sucks and was developed sloppily. I think they should face consequences for their sloppiness, I just don't think they will, under current legal frameworks. At best, maybe people will vote with their wallets, going forwards.)
On top of that there are companies that had failures of their own in their recovery procedures. But even with good procedures this can be a significant outage because it is not trivially reverted and would typically affect many configurations that are redundant for many other failures.
2. Your IT department shouldn't buy a product that doesn't give you control on when updates are applied.
These are 2 huge security failures from your IT department.
You can have automatic security updates with delay between non prod and prod environments so that you can detect failures or possibly intrusions.
Not even remotely correct.
Most computers that were affected by the fault needed physical remediation via safe mode boot to fix the issue because they were not able to download a fix because of being stuck in a reboot loop. The understanding is that for most cases, the fix needed to be applied by an IT technician dispatched to physically access the computer.
A week or 168 hours later, there are still many, many computers out there that remain bricked by this fault because it is so heinously difficult to fix.
"Neither the offerings nor crowdstrike tools are for use in the operation of [...] direct or indirect life-support systems [...] or any application or installation where failure could result in death, severe physical injury, or property damage."
I could imagine this was not the case if you had to physically access remote servers, or didn’t have access to bit locker recovery keys
I spent a weekend and abused a c&c infrastructure server to fix the clients and remove the flaw and malware. I see very little sophistication there.
You start with correct design.
The system has a root of trust (ideally you skip the insane level of complexity that is Secure Boot + TPM and use something simple, testable, and verifiable — this isn’t actually that hard). Only authorized images will boot, and, more importantly, nothing else on the network trusts the machine until it proves it’s running the right image.
Then you make the image immutable. Want to edit a system file? You can’t. Maybe in developer mode you can edit an overlay.
All configuration is stored in a designated place, and that configuration is minimized. A stock image from the distro vendor has zero configuration, so there is no incomprehensible soup in /etc to audit. Configuration is also attested.
Persistent data is separate from configuration. All persistent data is considered suspect. Any bug that allows malicious persistent data to compromise anything is a blocker, including corrupt filesystem metadata.
A root-of-trust attestation has limited lifetime. The system forcibly re-verifies periodically. This either means rebooting or doing a runtime “dynamic root of trust” attenuation. The latter is complex.
Complicated messes like kernel “lockdown” and the stock Secure Boot signatures have no place. Usermode root and the kernel are approximately equally trusted. SELinux is barely necessary, if at all, unless the actual user code wants it to control access to persistent data. But there are simpler, better schemes that are easier to reason about.
Sadly the industry doesn’t think this way. I’m regularly surprised that Apple hasn’t gone in this direction more aggressively than they are with their MacOS products.
I'll strongly disagree on selinux, I have seen it work in practice to defeat attackers many times, that provide features that seccomp and cgroups etc do not.
If your system is a flight information display, then you may well have two userspace processes that do anything of significance: the display manager and the actual app. There is no persistent state. At this point, SELinux is purely overhead and extra attack surface — what would it even protect.
If your payload is a container (database server, microservice, whatever), and you’re doing some form of best-practice volume management, then only the database’s own data is mounted for it. SELinux is a real PITA to get working in a context like this, and it’s not really clear what it would add. (Okay, maybe you get fancy and use it to restrict what can talk to the microservice. Or maybe you use network namespaces.)
If you’re running a desktop or a more conventional server setup, then, sure, MAC policy has its place.
Imagine if they got access to local code execution... Binding to sctp protocol would instantiate the whole protocol in kernel. Effectively opening up whole new attack vectors. I can't see any other techniques (other than selinux like AC) that enables this kind of attack space reduction as easily.
I am aware that you can blacklist modules,etc but this is just one of many examples.
While SELinux can be set up somewhat orthogonal to the running system. OTOH systemd should make it easy to confirm every service process
In the specific example of sctp, one can turn off loading of modules at runtime entirely.
Malicious code doesn't need to run as root in order to completely destroy a business.
Not to mention, all of the things you describe are very nice if the kernel is perfectly secure. But it's not, so it's always possible and even likely that compromising any user on the system is equivalent to compromising root. And if you compromise one system, you can then exploit bugs in other systems' kernels that might allow RCE through well-crafted packets or other exploits that gain access without running through any user-space code that might validate those attestations.
And finally, when a vulnerability is found allowing such exploits, you now need to update all of these readonly systems - and this happens at least once a month. Do you go with a USB stick to each of 10k systems on five continents to update them?
This kind of smug "I know better than the rest of the industry, security is easy if you do things my way" is rarely productive or applicable.
How do you do this on modern commodity hardware without secure boot?
Or do you assume something in the category of embedded systems that allow to blow some efuses to get similar trusted boot?
It’s not necessarily easy without Secure Boot, sadly. The actual straightforward solution is boot ROM. It would be nifty if someone made SD cards, eMMC devices and such meant for this use case for independent use. Most Android vendors manage to use boot ROM.
Depending how advanced the attacker is, check the executing binary maps back to the actual expected name and location on disk. Make sure the executable and libraries used at runtime are the correct ones matching hashes of known good qualities.
Ensure the process tree structure has an expected structure, ie "bash" isnt starting a process called apache.
Make sure the selinux policy is correct for the process that is running. (I have no idea about apparmor)
Check to see if its linking to the expected binaries, that its not using 'hidden' files (starting with a dot or directory with a dot), or deleted files.
Confirm that the process is opening sockets and files that. you expect it to (ie, apache shouldn't open files that are outside its configuration directive).
The process should not be making outgoing socket connections unless it is a client.
It should not be running with capabilities(7) that it does not require. It should not be executing from a setuid binary.
Check the process name, quite often attackers rename the running executable, so you'll see /proc/pid/cmdline renamed with a bunch of null bytes at the end.
Some malware has 'anti debugging' tactics, ie, they have traced themselves to prevent you tracing them, you can find this as one of the lines in /proc/pid/status iirc.
There are more, but thats the few off the top of my head.
> or a kernel.
This is a MUCH harder problem, because attackers can always disable any security mechanism assuming they kernel code execution. However, assuming they are not too focused..
If the system is booting in secureboot mode, it should be enabled, and no extra / unused / out of date kernel modules loaded.
I know that code injection at the memory level means that attackers can inject unsigned code, so in this case you would want to periodically sample the code and ensure that execution context would only have the processers EIP in known areas where the kernel would map executable code. You could do an additional check to see if the areas are mapped by userspace processes (it might be too late) so you can find offending attackers.
If the host is virtualized, this becomes easier to do and mapping and comparing memory from the guest kernel for the executable code sections means that its harder for an attacker to work around by being able to disable a mechanism.
Usually attacker kernel exploits do not persist long temr in kernel space, (they abuse kernel space to allow for userspace privilege escalation ie make a binary setuid or modify permissions on a /dev/) because the longer they are there the more likely they are to panic the system.
Some of the more advanced attacks I have seen are from people uploading system kernel panic images, where I have a 'snapshot' of the running system and can work around attackers mitigation techniques.
> There are more, but thats the few off the top of my head.
And that is probably like 80% what EDR product will be doing, checking that the code that is executing is trustworthy and not doing some weird unexpected things.
Stock Windows actually implements something along these lines, called "protected folders" or similar. It's inactive by default (meaning every program can access every folder). It's quite easy to define a list of "protected" folders. But the implementation is quite stupid: if a program asks for access to one of the folders on the list, you can either refuse, or allow it to access it... as well as everything else on that list!
https://www.theregister.com/2024/07/22/windows_crowdstrike_k...
They can come up with something more shielded as Apple has done, they just have to eat their own dog food and can't make an exception for defender. That's all.
Blaming the EU here is pure spin.
If you want to hook process execution or file access, you're writing a kernel driver.
They could easily have added a userspace API if they wanted to. It could have existed side by side with the kernel option, as long as they keep using that for Defender too. Only once they stop using kernel access in their own security products can they force the other vendors to use a new API, which makes sense. Otherwise they'd use it as a sales bullet point ("Our product has full system access, others don't"). Which would destroy the antimalware market. The US benefits from this too.
for a lot of things on Windows there isn't anything like eBPF (yet, it's wip, but likely will still take quite a while until it's usable)
the EU spin would only work if CrowdStrict is fully incompetent like a lot of people want you to believe. I.e. they don't do any testing, don't do any config validation and doesn't know what they are doing at all
but that simply isn't true at all
This doesn't mean that they didn't act negligent, as far as we can tell they relied on some data format validation instead by their server + signing (or something similar) instead of _also_ having robust parsing and that is enough against best practices to be called negligent. And there were other points which bubbled up in the last week which point to other negligent behavior unrelated to the bug. But company ending up with some negligent behavior and them being fully incompetent are very far away, let's be honest most IT companies today have ended up with some negligent behavior they have lite direct/short term/fast feedback motivation to fix (hence it doesn't happen)
It has not, however, proved enough to fend off different real world problems like ransomware.
Hence, the market for 3rd party solutions that are more aggressive. And to keep up with real world threats, they have to update often. And have to run at high privilege levels. So now you have the situation where those third-party solutions have the ability to create a bsod and/or a boot loop. Which should mean that they have a very well thought out way to roll out updates.
Since then I just use Defender and never had any trouble or a virus or ransomware. Only issue is that sometimes the antimalware service takes a lot of CPU.
This is not neccesarily a good thing for MSFT as it will 100% trigger regulator rage in the EU.
https://www.statista.com/statistics/917405/worldwide-enterpr...
https://www.microsoft.com/en-us/security/blog/wp-content/upl...
A screenshot of one comparison from Mitre:
You can do more of them here: https://attackevals.mitre-engenuity.org/
It's not a huge difference, but there's a difference.
Also, I have no relationships or investments, etc. Not shilling.
Edit: Also, that url slug from imgur. Heh.
Also, you don't want operating systems to provide this actual EDR program. They need to provide the facilities for EDR vendors / creators to tap into and do their work properly. You don't want a butcher to rate their own meat... you want a third-party to do this. As Example: MS Defender is totally rubbish (general sentiment for a lot of people in security, hence they run falcon or cortex XDR etc.) at defending Windows.... and it's by Microsoft. They should focus on building an auditable OS and let auditors do the auditing...
The best thing imho is a tool like CSF but integrated with network appliances (which CS doesn't do i think), which is where the strength of such tooling really comes together, correlating network data / behaviors to endpoint behaviours and having a full 'causality chain' of processes / systems and network traffic invovled in an attack.
And you are right on the balance of security being dramatic. using crypto is still hard as ever, and allowing external parties to interact with your users is just impossible to do right (let alone have users in the right awareness mode). This last is a problem of security industry imho, making tools so difficult.
Someday maybe rather than EDR tools and firewalls, cybersecurity companies will deliver 'secure business services' which are easy to use, userfriendly services that are secure by default. - maybe in like the year 3042.
Sure, it may not be the best, but most vendor solutions aren’t either. Case study: crowdstrike.
E.g. downloading a file and running the contents as code, or uploading/encrypting all files you have access to.
Crowdstrike and Defender handle those possible but suspicious actions.
As to why windows is not more locked down- that's on the shoulders of the admins. But out of the box, you are right, it is to permissive. But apparently users and management like it that way.
They are. It doesn't, y'know, do anything. It ticks the box for your auditors and occasionally makes your computers stop running, which is par for the course in regulated environments.
I think one massive difference between CS and AV is also, you don't expect a human to be in the loop because it would be too expensive. Nor would it be feasible for consumer software because of privacy.
Also even within this small niche, the solutions are very heterogeneous and make little sense for single boxes - in fact may even be designed to run on a network level.
For example, by comparison, Linux is in the stone age here.
Do you even need AV if untrusted code can't run in the first place?
* Application whitelisting - with just bare old AppLocker, Windows can be configured to only allow execution of trusted executables, DLLs and scripts by path, hash or software vendor (digital signature). Now, technically AppLocker is not a security feature, i.e. a hard security boundary.
The next level functionality, Windows Defender Application Control (WDAC) [1], however, is. I believe Microsoft was offering up to a $1M bug bounty for WDAC bypasses?
With WDAC kernel mode code integrity enabled, only trusted digitally signed kernel modules can be loaded into the OS kernel [2]. WDAC user mode code integrity provides the aforementioned protection AppLocker provides.
With AppLocker / WDAC enabled, the OS built-in script interpreters (Windows Script Host, PowerShell) either refuse to execute unsigned scripts completely or operate in restricted mode with reduced functionality.
- By comparison, Linux only has fapolicyd which is only supported on Red Hat and can only rely on path-based rules because binaries are not directly signed on Linux. None? of the common interpreted languages (Python, Perl, Ruby, Bash) on Linux support digitally signed scripts and locking down interpretation.
* Authentication material protection - Windows has Credential Guard [3] for protection of authentication material - Kerberos tickets and other material are placed in a separate container protected by hardware virtualization [2] and accessed via RPC so you can't dump process memory to compromise them. Even kernel level compromise is not enough.
- By comparison, Kerberos tickets on Linux reside as files on disk, SSH user & host keys reside as files on disk and loaded into sshd/gpg-agent memory, x.509 keypairs reside as files on disk & process memory etc etc. Wouldn't it be nice to have them protected somehow? To my knowledge, nothing exists for this on Linux.
[1] WDAC - https://learn.microsoft.com/en-us/windows/security/applicati...
[2] VBS - https://learn.microsoft.com/en-us/windows-hardware/design/de...
[3] Credential Guard - https://learn.microsoft.com/en-us/windows/security/identity-...
I have always wondered about that; there has to be a more secure control method for those secrets.
(1) - Why is a crutch like "anti-virus" software needed? Essentially trying to reactively cat-and-mouse hostile software that the OS has let execute on the computer.
(2) Why doesn't Windows provide AV?
Question (1) is more interesting - and (2) is addressed by other comments.
I think both MS and their customers have very seldom prioritized security over even small compromises in functionality. We loudly blame MS but they are the vendor MS customers deserve. While it's not a democracy, there are parallels to the popular sport of blaming politicians for eg not doing hard choices against climate change while holding the voters innocent.
Maybe some day someone will write an OS that is "fully secure" and then they'll be able to confidently run a system whose users can confidently click a link in an email, download an .exe from there, and run it, without fear of losing or leaking a single bit of data. That day is definitely not here, and until then, we all do the best we can through education and security appliances.
Secure operating system designs tend to simplify and take away stuff rather than add more bells and whistles.
I think SeL4 might qualify, but that can only realistically be used for embedded applications, it doesn't have, at this time, many of the features you'd need to build, say, an HTTP API server for it.
On the research side there's lots of stuff. Singularity, the various capability based systems, Qubes (granted more towards the adding-features dimension), etc.
Also, how about malicious scripts that I convince you to explicitly give execute permissions to and run? How about Git repos that I convince someone to clone, compile, and run, that have malicious code?
Signature-based heuristics can help protect from all of these things that the OS is powerless to help against with only traditional security measures.
The issue is that it's the only "level of defense" which introduces arbitrary non-deterministic behavior. An executable which correctly follows all the APIs as documented and implemented, and which does nothing malicious, might arbitrarily be denied or even erased, and this behavior changes daily or even hourly due to factors outside the control of the computer's user. Even ASLR, which uses non-determinism in its implementation, doesn't cause non-deterministic behavior when an executable correctly follows the API.
And it's also a "level of defense" which famously causes frequent performance issues, to the point that "tell the AV to ignore that folder" is a common recommendation. I wonder how many gigawatts of electricity are wasted daily due to AV software slowing things down.
Finally, it's been reported several times that this "level of defense" is often poorly implemented, to the point that it can act as a backdoor to bypass other levels of defense. If you can compromise a parser running as SYSTEM, or even within the kernel, you don't have to worry about all the normal rules which prevents you from running code as SYSTEM or within the kernel.
People's dislike of AV software does not come only from some abstract purity ideal; it also come from plenty of negative experiences with it.
Of course, when you have a locked down system such as a server or an embedded device, the need for AV protection drops down significantly. But on a wide open system, there's really no alternative.
Finally, even when the OS does natively provide services like these (Enterprise versions of Windows do provide all the features I mentioned above), it's perfectly reasonable to prefer a different vendor for those solutions. Maybe people trust CrowdStrike's malware signature lists more than they do Microsoft's, for example: a good reason to buy CrowdStrike instead of using Windows Defender.
I'm not trying to defend CrowdStrike or Windows here. But I think it's obvious that there are many features that fall under the umbrella of security that you wouldn't want to build into the OS itself, and even when a version of them exists built-in, that a company may wish to source from a different vendor.
My understanding of law is generally UK based, but I'm not aware of legislation what would supersede a contract term limiting liability when the event that created the liability was one of general diligence/competence in carrying out the contract rather than relating to health and safety or some other area that is heavily legislated.
For that reason I'm unconvinced on the article's statement that this isn't just a "French Legal System" thing and that the same kind of judgement might be made in other jurisdictions.
If they willfully did not implement staged rollouts that look like negligence to me but ianal. You kill canaries for a reason.
most(all?) EU have laws which limit how much you can opt out of liability _no matter what you write into a contract_
while I'm not sure about the exact boundaries per country but I'm pretty sure that at least all hospitals, emergency call services etc. can sue for a non-negligible part of the damages that outage caused directly
private people which where harmed by not getting operations done in time most likely can also sue them for the full damages caused to them (through it's hard to assess the damages and it might need to be indirectly by suing the hospital and the hospital sues for more damages)
what you likely will not be able to sue for is the lost opportunity cost, the man power needed to fix it etc.
also my guess is that for a lot of cases which are not as sever as human damages or as indirect as lost opportunity cost a huge factor will depend on the degree of negligence judges believe happened. And here "negligence" isn't limited to the specific change which caused the bug but also if they kept they due diligence in choices of tooling, approaches, business processes etc. to reasonable minimize the risk. (like e.g. was their way of parsing configs inadequate/did it follow industry best practices (IMHO it doesn't seem so), or was it adequate to mark the driver as required to allow boot (else windows would have auto disabled it and then restarted) etc.)
I assume the year was meant to be 2024.
Is there a link with this incident?
"CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes", https://news.ycombinator.com/item?id=41030352
I think if the total liability for Crowdstrike is less than a few years worth of revenue, they'll come out unscathed because as I understand, they are still not profitable, their valuation is purely on speculation on future revenue. Their biggest paying customers still care a lot about getting compromised, it isn't just a box checking exercise like many have suggested.
And if France comes down hard on them, they may simply not do business in France.
From OP, in the OVH-case liability seems to override the contract / waivers when OVH was both the storage And backup provider and did not actively underline that this solution is suboptimal, in a situation where multiple data centers are physically very close. That's a chain of evidence.
For CrowdStrike, it is clear that the offering is to more mature counter parties (thus raising the B2B standard of evidence) and that CrowdStrike very essentially did not do / support staging, whatever. This is indeed bad industry practice, but one that can thought to be explicit from the start of the agreement. At least in my locale you either make explicit agreements OR industry standards are leading. We do not do industry standard X is pretty clear. Read the list in OP, replace CrowdStrike with Microsoft and then think of the international liability cases you've heard from where Microsoft was found liable for downtime, hacks and other issues.
Look, liabilities will always arise in such situations. But I expect only minor liabilities will arise. Mostly (AFAIK IANAL) the terms & conditions are applied in B2B-cases. This case is pretty obvious: you got what you signed up for. CrowdStrike with full scale access to your machines and no guarantees. On the other hand, Crowdstrike lost 125 billion in market cap. That's an indication of {liabilities + loss of future profits}. Pretty massive event for not being willing to do staging. But I expect it's mostly that CrowdStrike is tainted from now on. A friend of mine had a very bad stint as an employee of CrowdStrike recently and from what I learned from that case, I'm happy that the nature of the firm is somewhat more in the open now.
That would have been literally the headline I'd choose for the bug.
This is incompetence that in a just world would result in the corporate death penalty.
- Is it reasonable to grant such privilege access to a piece of software that ultimately is a black box ?
- Is it reasonable to put a Microsoft / Commercial / Closed source OS in critical infrastructure ? If not considered as critical, then “important” infrastructure ?
- Is it reasonable to have more than 70% of the computers/servers that run important infrastructure on the same OS / software ? How about the mitigation of the risks etc…
I sincerely hope that all of this CrowdStrike mayhem will push stakeholders to draw some conclusions and actions.
As I said in the previous thread: explaining to execs that giving root to someone on your machines means they have root is a very difficult concept for them to understand.
These standards are influenced not only by actual threats but also by lobbying from Endpoint Detection and Response (EDR) systems like SentinelOne and Crowdstrike. For instance, in 2021, the White House issued Executive Order 14028, which mandates the Federal Government to implement a robust EDR solution. Consequently, standards such as those from NIST and ISO27001 have increasingly emphasized malware detection and response.
When onboarding any large enterprise, you will encounter these requirements before the enterprise can proceed with procuring your service. This compels B2B organizations to implement this software to be successful.
^1 https://www.opensecrets.org/federal-lobbying/clients/summary...
^2 https://www.opensecrets.org/federal-lobbying/clients/summary...
This is the problem as far as I'm concerned. Industry "best practice" is "use the same thing everywhere"
A diverse ecosystem is the best defence.
You could run 100% FreeBSD and be hit by say a hidden kernel bug which occurs on Jan 15th 2027 when unix time goes from 1.7b to 1.8b (I've seen that code before where time is assumed to be below X)
If you run 50% FreeBSD and 50% Windows you will only lose half your service.
The hallmark of intelligence is to observe a situation and the structure of a system, reason about it, draw analogies with past experience and pre-emptively take corrective measures.
The stark truth is that we don't live in a "reasonable" world.
Poor governance, short termism, lack of transparency, incompetence, captured regulation, obsolete ideology etc. are not exceptions but rather the essence of how things "work".
The existential question is whether our demonstrable ability to achieve some learning will be sufficient to deliver solution on the face of increasing risks.
This is common enough in the corporate world and precedence in similar circumstances will come into play in various lawsuits.
Examples:
XYZ Security Guards: a third party physical security provider that hires people to watch and patrol buildings, assets, with access to keys, timetables, security logs, etc.
ABC Armoured Transport: third party physical transport provider for cash, sensitive documents, etc.
When AcmeCorp Inc. hire XYZ & ABC it's on the basis of reputation, contracts, and things generally not to do with peeking inside how the cake is baked (hiring records, etc).
Security is a complicated topic, and employees are also potential attack vectors. A system that is in the complete control of a malicious employee is a security problem for the company just as much as a system that was corrupted by an external cracker.
It's no different with software.
According to Microsoft its not but they were forced to. Interesting how the EU executive is now getting mixed up in this saga: https://www.euronews.com/next/2024/07/23/european-commission...
A hospital doesn't have, and couldn't use even if it did, the blueprints for an MRI machine or an old-fashioned iron lung. And those machines are built by commercial companies and contain plenty of trade secrets.
If anything, using open-source software that you maintain yourself in critical infrastructure is the more bizarre practice from a historical or industry-level perspective. Even in software, things like Solaris, IBM OSs etc. are much more common than OSS. And even when using FOSS, a commercial distribution like RHEL is far more common than using your own Linux.
Even something like a fountain pen is used as a black box, I'm not even talking of anything truly complex. Even the buildings we work in are black boxes that we get from third parties, not to mention all the systems powering and heating or cooling them.
https://www.abc.net.au/news/2024-07-26/crowdstrike-gift-card...
I almost feel sorry for CrowdStrike.
But you would not get away from $mil of damages with a $10 voucher. The courts are not dumb and don't work like that.
Maybe now ClownStrike will start testing it properly, hopefully thereby fixing the stability and other issues.
Personally, I don't expect this to make much of a difference, if any.
While you're probably right, I'm hoping ClownStrike's court results so absolutely dwarf their insurance coverage that it's nearly company ending.
ie something to actually get them to improve things, not just generate empty PR platitudes: https://www.youtube.com/watch?v=SiL2AjOtjZI
So, I expect this to be spun somehow along the lines of "sure all our boxes were down, but look, you've brought them all back up, didn't you? Now think about all the bad guys this protects us against! Of course the risk was worth it!". Also, "everybody does this! we couldn't have known!"
"security people" are scared shitless of the whole "the world is ending! there are threats everywhere!" discourse that vendors peddle. The less technical, the more scared they are.
Many of these people don't really understand what they're talking about and what compromises their decisions actually imply. Losing a day of work is simply dwarfed by "all your data is gone!".
I'm pretty sure that all their resources are allocated to lawyers right now and their managers try really hard to gaslight their customers by telling them it was not that bad, they came up with the fix really fast (ignoring the fact that the fix was not possible to be applied) and so on.
in many countries there are very strict limits on excluding liability for negligence
Right, but other commenters call it the best EDR out there; so it is really hard for those of us outside the loop to understand what the hell is going on. Is CS, or any other EDR, actually preventing attacks that would pass through if absent? To what extent? Where are the numbers? Who audits CS code? I have seen no real data, only assertions.
I guess the people saying that are security folk who look at things from a high level place of some sort (?). Because they don't seem aware of (or don't care about) the many problems it causes on the servers it gets deployed to.
As we've now had amply demonstrated. Globally. ;)
To whoever does this I have only one quote from Jaws:
You go in the cage, cage goes in the water, you go in the water, shark's in the water, our shark. Farewell and adieu to you, fair Spanish ladies. Farewell and adieu, you ladies of Spain.
No-one is seriously claiming CrowdStrike did that.
Eh, parts of this article aren't very reasonable. Even if they did a buttload of testing, it only takes one failure in one part of the chain (near the end).
They didn't test something they should have, sure, but obviously they didn't do "no testing whatsoever"
...based on the OVH precedent
Corporativism in US is a thing. Companies can brick hospital systems killing patients, drive self-driving cars and run over people but don't get sued, and if they do, they settle for very little.
Just look at the recent Boeing incident where people were killed, the company clearly misled the US authorities and settled only a $0.5B fine.
Those companies in those scenarios should pay the fine that they should ($20B+), and if it means the company would go bankrupt, do it and form a new company diluting the previous shareholders.
Without doing this, shareholders and CEOs will have the incentive to carry on with their unfair practices that leads to dead people and deadlocked systems.
What is the advantage over just fining? We keep trying to reïnvent the fine, which is great for those who would otherwise be fined.
It feels like US can hit as much as they can EU companies, but EU needs to create a whole new regulation to slap a $1B fine in a 2 trillion company from the US.
The problem is when you fine a company, they will just turn around and offload that cost to their customers. Which in this case is the US government in a very large way. Boeing will make their part in the SLS a few billion more expensive again to offset it and even gain some profit. The US government are just fining themselves.
Fines just aren't an effective deterrent for companies. They should go back to imposing personal sentences on their leadership. But this is really unpopular because these guys are so well connected. So basically nothing is done and everyone goes free.
See what VMWare did after the broadcom takeover. They did exactly that.