"Any sufficiently bad software update is indistinguishable from a cyberattack"
twitter.com
twitter.com
Centralizing power is asking for abuse.
Not particularly. Whatever you do, it can be (will be?) abused. Centralizing power means that a centralized power abuse will happen, like in the case above. In a distributed power system, an attack could occur that targets most entities successfully, but they won't respond efficiently because of lack of centralization. Like computer viruses in the old days.
And you are still left with the GPU drivers.
It'll be interesting to watch how nVidia responds to this move. Their drivers are black magic at the moment.
A big problem here is that Microsoft has normalized the idea of rando third-party software having access to ring 0 and things running at the kernel level. This is one of the reasons many Linux people argue against opaque third-party blobs running in the kernel. This should not be as routine as it is in the Windows world. Apple wisely ditched kernel extensions.
1: EDIT: Ideally, I'd rather Microsoft not have access to remotely update my system, either, but that ship seems to have sailed long ago.
Personally I’d go for diversity over any specific solution. That’s a rare thing in “enterprise” world. I suspect companies with strong shadow IT which provides business value for far better than those top down enterprise led ones.
Back in the day, you could install windows XP and be infested in minutes https://en.wikipedia.org/wiki/Blaster_%28computer_worm%29
Ring 0 is bad but malware do get SYSTEM all the time, they can inhibit booting at that point just the same.
Yes, well, that's because we have built much more resilient systems, of which anti-virus is a substantial part of. You could start with the Morris worm, but a more recent example would be the Stuxnet cleanup efforts (largely done with AV).
AV is (thankfully) becoming less and less relevant, but it's still a valid layer of defense for many platforms.
Pretty much every common modern-day security defensive is reactive, in that you can point to a point in time where we didn’t have it, and can pretty easily see the consequences of that.
“Any security product which has a rootkit, remote command and control and egresses data is indistinguishable from malware”.
Any security that isn't done layer by layer in depth must be "tacked on" and try to know everything about a system at once and adapt in situ. Which is of course impossible on any given machine, you say. "But what if we leverage the power of the crowd?" said someone.
All these “blah blah blah is indistinguishable from malware” things aren’t profound, smart, or even witty. They’re spouted by the peanut gallery that has the luxury of not being responsible for deciding whether or not to use one of these systems.
Needing to explain to techies that ‘informed consent matters’ speaks to an utterly saddening stereotype.
There isn't any. None that is meaningful. Sure, you can trick someone into 'signing' something, out of desperation and confusion. But the average person has no capacity. It's not that people are stupid, they are simply not informed nor are they ever entreated to anything that even looks like an actual contract. This is the colossal elephant in the room of digital tech.
Those teams 100% have the capacity to make an informed decision.
Not sure about that. Groups of professionals don't appear better at navigating this space than individuals. I'm sure you've sat in such agonising meetings too. Common experience: They're hellholes of group-think, risk aversion, inertia, legacy constraints, resistance to change, pressure to reach fast decisions, duress or undue influence from salesmen and 'partners'.
Have you ever seen a company of any size actually sit down, open-mindedly weigh up a real and serious evidence-based long term security plan built around risk analysis, a full network and service overview, with all real software options on the table and all stakeholders present. Companies made up of well educated people with impressive job titles are as vulnerable to pitfalls and shortcuts as anyone else. They just operate, and fall victim to scams, on an organisational scale. Crowdstrike and other protection rackets are a way to make a problem go away, not to face its complexity head on.
FedRAMP-authorized: CrowdStrike's cloud-delivered solution meets the strictest federal standards.
DoD IL5-authorized: CrowdStrike's solution is approved for use on the Department of Defense’s (DoD) Impact Level 5 (IL5) systems.
JAB High-ready: CrowdStrike's solution is validated, tested, and certified for use in hybrid, multi-cloud environments, meeting the Joint Authorization Board (JAB) High requirements.
OMB Memo M-22-09: The Office of Management and Budget (OMB) mandates a Zero Trust security approach for Federal Civilian Executive Branch (FCEB) and DoD systems.
OMB Memo M-21-31: The OMB directs investigative and remediation capability improvements.Based on other comments it can run as kernel module or as eBPF filters on linux. So I guess to them it's a less invasive/more power tradeoff which they'll take whenever it's available.
https://news.ycombinator.com/item?id=41030352
eBPF can cause Linux kernel panic
> The program does not crash or otherwise harm the system.
So while anti-malware might have some merits, on balance much of it would be a detriment to security, from this perspective.
People with this perspective are feeling spectacularly validated today!
As often, it's wise to have a nuanced view of course.
it's normal that from time to time someone will fuck up, but if they would rollout this update to only 1% and see that nothing that updated is coming back and bailed out this would be literally 100x less disruptive that what happened
• Developers building binaries on their own local systems (identified via metadata) that are then pushed out globally. Inference: 1 xz-insider = global ring0.
• Customer success teams sending 50 page reports on why they can’t do staged rollout (“respond to 0days”), but reality is they’re unwilling to invest in the eng infra to do so.
• When they broke Linux endpoints a few months ago, “post mortem” was ignored and nothing remedied.
Losing billions of dollars in stock market cap, recovery costs, and possible law-suits tends to focus the mind on these things.
Still up 20% ytd from an already high valuation, so they have a long way to fall.
If they don't fall all the way to zero (spoiler alert: they won't) they got off easy.
They may have not done any testing at all because they were adding a named pipe to be monitored, kind of like adding a domain or ip to be monitored.
There is a story from early 2000s, when some guy from East Europe did network administration, in exchange for fast connectivity and warez hosting. "Customers" had no idea, and he kept their servers running for a few years longer.
Allow it but make it hard. Put lots of warnings. Require enrolling custom keys.
In the end unless system is entirely locked down by the vendor without any compromise that will happen... And most of the time these customers will choose something else that is not entirely locked so they can have their security thingy...
Like, for real - as in, Debian would be happy to include it in their main repo? Or lip service? (Assuming you meant their graphics drivers)
Not upstreamed yet so I’m guessing it will take a while to reach Debian.
I can build a webcam with an rpi that will play CS better than any human and only communicate with the gaming PC through a mouse and keyboard.
Anti-cheat systems need to look for movements that are super human, no rootkits required.
That won't be effective for wallhack/ESP-style cheats.
Just imagine if a web page had to send a network request to control a toggle or to show the contents of an accordion; that's what the game would turn into if this were to be done.
So many idealists looking to make this a “closed-source bad!!!” thing and in the process muddying the waters enough to take attention away from remedies that might actually work.
All while they sit there getting paid $500k/yr to write closed-source software at FAANG or a startup, which to them is Technically Okay because they work on some sort of SaaS product, thereby alleviating them of the economic realities of Everything Being Open-Source.
Most definitions of open-source (e.g. OSI) include rights to modification. Without modification and distribution rights, programs are normally referred to as source-available instead. This is a common complaint on this very forum when companies misleadingly market their SA code as OS.
Having to manually push and pop stack is not something to proud of, only do that out of necessity. There's a very reasonable option between C or C++ and Python (or whatever is deemed slow). Using C doesn't prevent you from designing a shitty application that uses shitty algorithms.
"Ooohhhh I get it, this pointer is no longer valid because the array have been reallocated because that function have this side effect after that commit", he said as he gave out a small smirk at the hilarity of this bug. I wonder if he would find it as amusing if his coworker regularly rolls his sleeves and randomly regularly writes assemblies directly in the source code because compilers aren't smart enough to perform a particular parameter passing optimization that he really loves to do.
A slight aside: Rust is typically held up as the obvious safer alternative to C/C++, as if Ada never existed.
[0] https://stackoverflow.blog/2024/03/04/in-rust-we-trust-white...
> The teams didn't know they were in competition with one another, but the much less experienced C team completed their solution so much faster than the Ada team, that they just dropped the Ada effort
I realise you've given an abbreviated account, but it implies that:
1. No regard was paid to the relative quality of the 2 solutions
2. They ignored that initial development effort tends to be outweighed by ongoing development and maintenance
3. Many programmers are already familiar with C but not with Ada
The reboot part happens because the system is assumed to be in a bad state and allowing it to continue would possibly corrupt data, or in the worst possible case execute exploit code.
This panic handler runs in the same privilege as the faulty driver and can itself be prevented from running correctly. Notably file system drivers are required to function correctly to write the memory dump. If they, or filter drivers attached to them, also fault, well, fun times.
You can have faults in an interrupt handler too, for example trying to access paged memory in a page fault handler. That'll trigger a double fault handler and if you fault in that, the processor will perform a reset and not bother even notifying software. Luckily the double fault handlers and other such cases are usually solely the preserve of OS vendors.
I have no particular point except to illumate what's going on and that processors (in this case x86 terminology is used) and that actually recognizing and aborting from an invalid state is exactly what's happening here and what rust memory safety does. In spite of the disruption that's better than silently corrupting data.
Here's Linus's commentary on that:
https://lkml.org/lkml/2021/4/14/1099
> I think that if some Rust allocation can cause a panic, this is simply _fundamentally_ not acceptable.
> Allocation failures in a driver or non-core code - and that is by definition all of any new Rust code - can never EVER validly cause panics.
Panics are not acceptable in countless contexts. Plenty of things need to be written to keep working through entire categories of errors. The casual attitude of Rust developers towards error handling is one of the many reasons people have trouble taking it seriously. Reliability and robustness is generally more important than language memory safety for almost all contexts.
But, in the incident in question, the code is fundamentally not correct. Spatial memory safety violations, or in plain English "trying to call functions or use data that isn't at addresses your code or data lives at" fundamentally is an error. There's a missing part of the state machine to detect and stop before just exploding. In userspace this is a segfault and your process dies. In kernel, you get a bugcheck and the whole system reboots.
There are scary alternatives. The first, in kernel, is that you suppress all invalid writes and allow the errant code to keep writing, until it hits some other data. The system stays up, but you have out of control data writes so who knows what that's doing.
The second is that the execution flow of the process can be hijacked, i.e. Sergey Bratus' weird machines, or in plainer language, owning kernels in critical infrastructure. This is usually undesirable.
Panics in a real-time system or a kernel are quite possibly not.
Clearly, you can write shitty software and this is also far too easy in C, but there are also many tools and techniques one can use to write reliable software in C. Rust is also a tool that can be used to write more reliable software, but it is not clear to me why this should suddenly make the big difference. People who want to write bad software can also sprinkle "unsafe" everywhere, and I would guess that this one happens a lot more once Rust is adopted more. And in the CrowdStrike case, Rust's default behavior to panic might not have necessarily prevented the problem.
Having said this, regarding security against malicious hacking, memory safety is indeed one important component and Rust clearly has an edge there. In terms of general software bugs, it switching to other languages than C/C++ is certainly not a magic bullet and I often doubt that it would even an improvement in most scenarios.
After several years of efforts. I have written several apps in JavaScript, Rust, C++, Python. Of these, only Rust is the one that I can safely NEVER look back and be satisfied. Everything else just left me wondering if I have missed an enum or something stupid like that.
> People who want to write bad software can also sprinkle "unsafe" everywhere, and I would guess that this one happens a lot more once Rust is adopted more.
People actually _dont_ like to use unsafe (some people do, surprise, surprise, they are usually C folks). Most people that I see usually wants to get as quickly as possible out of unsafe. Why not? Believe it or not, people actually don't want to deal with tagged enums, or parse JSON themselves, or care about how do i properly set up SIMDs. They just use correct APIs so they can sleep at night instead of staring at stack traces.
Rust is not perfect, true, but it is way, way better than C. How many software have been written in other language, and then the authors decided to write it back in C? If this does happen, it is usually something that is very well defined already. I haven't heard anything significant recently. I think there's a place in C as it provides a very stable and reliable API.
> switching to other languages than C/C++ is certainly not a magic bullet
its definitely a magic bullet that kills a LOT of problems tho? No one is saying it fixes everything. If we keep using C or C++ for the next 1000 years, we will have the same sets of problems. Things will NEVER improve.
And yes, it happens that stuff is written in C. But it is not discussed every time on HN such as the many "XY written in Rust" articles, because the the later apparently is still newsworthy.
When the Clownstrike incident occurred, the phrase "fucking for virginity" popped into my head to describe what software like that does in the name of "security".
E.g. supply chain attacks have become a hot topic in the last few years. This event suggests that your threat model for supply chain attacks should include catastrophic vendor cockups.
Who says that?
Probably a wise lawmaker.
Who's law is this?
Leigh's Law?
Sounds right to me :)
There are bad software updates that are not malicious, just inept, or an unfortunate accident that cause havoc. I think the Crowdstrike event was such a mishap.
And there are plenty of software updates that are plain malicious but hiding behind the "legitimacy" of an update. I'm thinking here about Amazon deleting the 1984 book, or printer 'updates' that lock-out third-party ink etc. These are really violations of computer misuse acts and ought to be prosecuted - because they are indeed vandalism indistinguishable from a attack.
Maybe they're worse than a cyberattack, because they are harms that abuse a privilege.
Because companies have not been prosecuted but allowed to get away with this sort of crap for decades were in a sticky situation now.
There's a whole spectrum of intent between sincere security updates that go wrong and spiteful for-profit sabotage. People need educating that if you allow anyone remote access to your computing property, no matter what their credentials and bona-fides, they are in a position to massively abuse that trust. Just because someone sold you some hardware or software doesn't mean they continue to have your best interests at heart or any rights to interfere with your property.
All software and hardware should by law come with the ability to lock out the original vendor, supplier and to reliably stop egress and your device from "phoning home with telemetry".
The customer is buying a product not a relationship.