Hacker News Reader

Comment by eitland | Hacker News Reader

Some Canonical guy I think many years ago mentioned this as their sales strategy a few year ago after a particularly nasty Windows outage:

We don't ask customers to switch all systems from Windows to Ubuntu, but to consider moving maybe a third to Ubuntu so they won't sit completely helpless next time Windows fail spectacularly.

While I see more and more Ubuntu systems, and recently have even spotted Landscape in the wild I don't think they were as successful as they hoped with that strategy.

That said, maybe there is a silver lining on todays clouds both WRT Ubuntu and Linux in general, and also WRT IT departments stopping to reconsider some security best practices.

View on HN

Except further up this thread another poster mentions that CrowdStrike took down their Debian servers back in April as well. As soon as you're injecting third party software into your critical path with self-triggered updates you're vulnerable to the quality (or lack of) that software despite platform.

Honestly your comment highlights one of the few defenses... don't sit all on one platform.

Hopefully they won't botch the update for two operating systems at the same time. But yeah. Hope.

Yeah, I see a lot of noise on social media blaming this on Microsoft/Windows... but AFAIK if you install a bad kernel driver into any major OS the result would be the same.

The specific of this CrowdStrike kernel driver (which AFAIK is intended to intercept and log/deny syscalls depending on threat assessment?) means that this is badnewsbears no matter which platform you're on.

Like sure, if an OS is vulnerable to kernel panics from code in userland, that's on the OS vendor, but this level of danger is intrinsic to kernel drivers!

True; except here's what's baffling:

CloudStrike only uses a kernel level driver on Windows. It's not necessary for Mac, it's not necessary for Linux.

Why did they feel that they needed kernel level interventions on Windows devices specifically? Windows may have some blame there.

Yeah, I think your point is totally valid. Why does CrowdStrike need syscall access on Windows when it doesn't need it elsewhere?

I do think there's an argument to be made that CrowdStrike is more invasive on Windows because Windows is intrinsically less secure. If this is true then yeah, MSFT has blame to share here.

I don't know about MacOS, but at least as recently as a couple years ago crowdstrike did ship a Linux kernel module. People were always complaining about the fact that it advertised the licensing as GPL and refused to distribute source.

I imagine they've simply moved to eBPF if they're not shipping the kernel module anymore.

I haven't looked too deeply into how EDRs are implemented on Linux and macOS, but I'd wager that CrowdStrike goes the way of its own bit of code in kernel space to overcome shortcomings in how ETW telemetry works. It was never meant for security applications; ETW's purpose was to aid in software diagnostics.

In particular, while it looks like macOS's Endpoint Security API[0] and Linux 4.x's inclusion of eBPF are both reasonably robust (if the literature I'm skimming is to be believed), ETW is still pretty susceptible to blinding attacks.

(But what about PatchGuard? Well, as it turns out, that doesn't seem to keep someone from loading their own driver and monkey patching whatever WMI_LOGGER_CONTEXT structures they can find in order to call ControlTraceW() with ControlCode = EVENT_TRACE_CONTROL_STOP against them.)

0: https://developer.apple.com/documentation/endpointsecurity

> Why did they feel that they needed kernel level interventions on Windows devices specifically?

Maybe because everyone else in "security" and DRM does it, so they figured this is how it's done and they should do it too?

My prior on competence of "cybersecurity" companies is very, very low.

> Maybe because everyone else in "security" and DRM does it, so they figured this is how it's done and they should do it too?

What DRM uses kernel drivers? And how do you plan to prevent malware from usermode?

> My prior on competence of "cybersecurity" companies is very, very low.

Dmitri Alperovitch agrees with you.[0] He went on record a few months back in a podcast, and said that some of the most atrocious code he has ever seen was in security products.

I am certain he was implicitly referring, at least in part, to some of the code seen inside his past company's own code base.

0: https://nationalsecurity.gmu.edu/dmitri-alperovitch/ ["Co-founder and former CTO of Crowdstrike"]

> CloudStrike ONLY uses a kernel level driver on Windows

Crowdstrike uses a kernel level driver ONLY on Windows.

CrowdStrike uses a kernel level driver on Windows ONLY.

Even better..

ONLY on Windows does CrowdStrike use a kernel level driver.

Apple deprecated kernel extensions with 10.15 in order to improve reliability and eventually added a requirement that end users must disable SIP in order to install kexts. Security vendors moved to leverage the endpoint security framework and related APIs.

On Linux, ebpf provides an alternative, and I assume, plenty of advantages over trying to maintain kernel level extensions.

I haven’t researched, but my guess is that Microsoft hasn’t produced a suitable alternative for Windows security vendors.

> AFAIK if you install a bad kernel driver into any major OS the result would be the same

Updates should not be destructive. Linux doesn't typically overwrite previous kernels, and bootloaders let users choose a kernel during startup.

Furthermore, an immutable OS makes rollback trivial for the entire system, not just the kernel (reboot, select previous configuration).

I hope organizations learn from this, and we move to that model for all major OSes.

Immutability is great, as we know from functional programming. Nix and Guix are pushing these ideas forward, and other OSes should borrow them.

It's interesting to me that lay people are asking the right questions, but many in the industry, such as the parent here, seem to just accept the status quo. If you want to be part of the solution, you have to admit there is a problem.

Non hardware "drivers" which cause a BSOD should be disabled automatically on next boot.

Windows offers it's users nothing here.

Sure, but note the sales pitch was to encourage resiliency through diversity. While that may not be helpful in cases where one vendor may push the same breaking change through to multiple platforms, it also may be helpful. I remember doing some work with a mathematics package under Solaris while in university, while my peers were using the same package under Windows. Both had the same issue, but the behaviour was different. Under Solaris, it was possible to diagnose since the application crashed with useful diagnostic information. Under Windows, it was impossible to diagnose since it took out the operating system and (because of that) it was unable to provide diagnostic information. (It's worth noting that I've seen the opposite happen as well, so this isn't meant to belittle Windows.)

Yes, I already heard one manager at my company today say they're getting a mac for their next computer. That's great, the whole management team shouldn't be on Windows. The engineering team is already pretty diversified between mac, windows, and linux. The next one might take down all 3 but at least we tried to diversify the risk.

You can also make rollback easy. Just load the config before the one where you took the bad update.

Of course that means putting the user in control of when they apply updates, but maybe that would be a good thing anyway.

Yep, these episodes are the banana monoculture [0] applied to IT. The solution isn't to use this vendor or avoid that vendor, it's to diversify your systems such that you can have partial operability even if one major component is down.

[0] https://en.m.wikipedia.org/wiki/Gros_Michel_banana

> don't sit all on one platform.

Debian has automatic updates but they can be manual as well. That's not the case in Windows.

The best practice for security critical infrastructure in which peoples lives are at stake, is to install some version of BSD stripped down to it's bare minimum. But then the company has to pay for much more expensive admins. Windows admins are much cheaper and plentiful.

Also as a user of Ubuntu and Debian for more than a decade, i have a hunch that this will not happen in India [1].

[1] https://news.itsfoss.com/indian-govt-linux-windows/

Windows updates can definitely be manual. And anyway, this was not a Windows update. It was a CrowdStrike update.

Oh, i thought it was tied to OS updates. So Windows is not to blame, if that's the case.

well, in another sense, Windows is certainly to blame partially. Several technical solutions have been put forward here and in other places, that would've at least limited the blast radius of a faulty update/driver/critical path. Windows didn't implement any of those. Presumably by choice and for good reasons: A tradeoff would be that software like crowdstrike is more limited in protecting you. So the Windows devs deliberately opted for this risk.

Or they never considered it, which is far worse.

Linux and open source also have the potential to be far more modular than Windows is. At the moment we have airport display boards running a full windows stack including anti-virus/spyware/audit etc, just to display a table ... madness

I'm a Kubuntu user that, seemingly due to Canonical's decision to ship untested software regularly, has been repeatedly hit by problems with snaps. What were initially basic, obvious, and widespread issues with major software.

Yes, distribute your eggs, but check the handles on the baskets being sold to you by the guy pointing out bad handles.

FWIW, while some people like Kubuntu, I have had much better results with KDE Neon.

Stable Ubuntu core under the surface, and everything desktop related delivered by the KDE team.

Thanks for the tip, I'm looking to jump ship to MX-Linux, just procrastinating the move right now.

Still haven't forgiven Ubuntu for pushing a bad kernel of their own that caused a boot loop if you used containers...

I’ll never forgive them for the spyware they defaulted to on in their desktop stuff. It wasn’t the worst thing in the world, but they’re also the only major distro to ever do it, so Ubuntu (and Canonical as a whole) can get fucked, imo.

Sure but if that Canonical sales person was successful in that, I'd almost guarantee that after they switched the first third they'd be in there arguing to switch out the rest.

Absolutely.

I'm just saying what they said their strategy was, not judging their sales people.

Many years ago an Ubuntu tech sales guy demoed their (openstack?) Self hosted cloud offering, his laptop was running windows..

Canonical in particular are no better, they do the exact same thing with that aberration called snap. They have brought entire clusters down before with automatic updates.

Seems like a reasonable strategy. Not just Ubuntu but some redundancy in some systems.

Ubuntu has unattended-upgrades enabled by default

Yes, but by default the only repo enabled for it is $(cat /etc/os-release)-security.

But CrowdStrike is security as well?

Yes, but it's not included in the upstream Ubuntu security repository. In fact, it's not available via any repository AFAIK. It updates itself via fetching new versions from CrowdStrike backend according to your update policy for the host in question. However, as we've learned the past days, that policy does not apply to the "update channel" files...

things are so interdependent that in this scenario you might now just end up crashing the system if either Windows or Ubuntu are down instead of just the one of them you chose

Reply on news.ycombinator.com