Hacker News Reader

Comment by jpgvm | Hacker News Reader

The correct lesson is to stop introducing more vulns into your systems by running "security" products. Crowdstrike was just an outage but could have just has easily been Solarwinds 2.0.

Crowdstrike is probably less bad than the alternatives that I have run into that are largely developed by very low cost engineers cough TrendMicro cough but even so, they aren't NT kernel engineers nor do they have the NT kernel release process.

Companies need to find ways to live without this crap or this will keep happening and it will be a lot worse one day. Self-compromising your own systems with RATs/MDMs/EDR/XDR/whatever other acronym soup needed to please the satanic CISSPs are just terrible ideas in general.

View on HN

Crowdstrike probably will be solarwinds 2.0, because last week demonstrated to certain nation states just how lucrative a vector it can be for them.

How likely do you think the next big one would be caused by mandatory bootkit anticheats?

Those don't run on nearly as many juicy corporate/government targets.

Probably runs in some dev machines through. Maybe it can infect a valuable FOSS dev and then do the xz backdoor without bothering with the Jia Tan persona.

From many of the comments I read, the reason is often regulation.

guess who's lobbying for that regulation

Misguided regulations are more common than not, that's the real damaging kind.

Probably more likely with tech when regulators are more tech-illiterate than the situation requires.

All kinds of other regulations can be misguided for other reasons too, lobbyists' jobs are primarily there to do the misguiding.

The regulation doesn't mandate these systems.

The regulation requires controls, there are usually lots of ways of implementing those controls.

However more often than not the people buying these piles of goop are more interested in the cheapest/fastest/neatest way of satisfying those controls, actual security be damned.

So when vendor comes a long and says "our product meets those controls! all you need to do is tell the auditor you use product $X and you are done!" well that is an awfully compelling argument when those are your priorities.

Of course every security minded person dies a little inside but that is just how the world works for now, no-one has been sufficiently punished for these bad decisions yet but it will happen. Not a question of if, just when.

The lobbying does matter though as they lobby for increasingly useless controls that are harder and harder to satisfy with actual security (read least privilege design, capabilities, sandboxing, etc) and audit logging and become increasingly about "how big is your ruleset in your SIEM or equivalent and does it contain a specific rule that we can show auditor for control $Y" (yay Splunk/SumoLogic lobbyists) etc.

Not to mention stuff like pushing for TLS MITM boxes (self-compromising all your outgoing encrypted sessions is beyond insane) etc. The modern "security" industry is an embarrassment sigh.

I am very interested in alternative ways of implementing controls to satisfy regulations; especially in ways that reduce, rather than increase, attack surface. I'd love if you could elaborate here, but links to documentation/technical articles are also welcome!

The same suggestions have been made over and over. Here's one from a decade ago:

https://ia801200.us.archive.org/1/items/SyScanArchiveInfocon...

It incudes glaringly obvious advice like designing your code around the principle of least privilege or sandboxing code that handles untrusted input.

Unfortunately, not much has changed since then.

I'm not up-to-date on regulations (or many other things) but decreasing the attack surface in a way that could give a direct percentage improvement could get serious.

If lots of people wanted to reduce their attack surface by about 90% or more they would already be unplugging their LAN cable (or equivalent wifi logoff) except when they have a strong need to be online to accomplish something that is not (or never was) possible to handle otherwise.

If that's not enough, then escalate until no-one without a desperate need to be online is ever connected, and then only during the most desperate times, if necessary.

Depends on how much isolation from questionable incoming code (whatever you did not specifically request) that you prefer to have how much of the time.

Ideally, all incoming code or downloads should be directed to safe storage and never executed until after the user has had a chance to deliberately scan it offline as many ways as they would like before attempting a test deployment.

Things just used to be a lot more ideal more of the time. So many kinds of business-as-usual once raked in the bucks without any online connection, have little or nothing more to show for it afterward, and now threats are eclipsing the benefits if any.

>the cheapest/fastest/neatest way of satisfying those controls, actual security be damned.

Cheapest, across-the-board is auto-patching when online, since so many systems are expected to be online whether it's good for them or not.

I thought it was plain to see it was moving in this direction consistently since the arrival of Windows Update.

That's the one big thing that initiated the possibility of mindless on-line patching replacing the previous deliberate, focused approach.

Every year since then fewer people even try to make software that is good enough to release under the original paradigm where it had full value, back when the need for patching would only occur on the rare occasion of true unforeseen bizarreness that could only be discovered after full release.

There's always been a real advantage when everything is more ideally suited to the task to begin with, and stays the same for years in a row except for when you really want it to change and know exactly what those changes are going to do in detail.

The need or preference for routine patching or updating is a dead giveaway for software that's probably not worth money to begin with.

I've done it, it's just drawing up documentation on why controls are satisfied another way and getting auditors to sign off on it.

It's such a PITA and CISO live by not doing anything risky that few teams can get away with it.

> The regulation requires controls

lobbied by the same security company

Essential Eight 'maturity levels' are vague in this regard, though seems to mandate it (at least 'application control'), at varying levels.

Seems like the only way third-parties could do it 'effectively' is with kernel-level access, though I am absolutely not an expert in this field (light years from it).

https://www.cyber.gov.au/resources-business-and-government/e...

Having been an end-user going through the adoption of one of these third-party application control and auditing systems, I won't say which, or for whom, I can definitely state it was painful. Said system broke quite a few business-as-usual processes, despite a months long 'learning phase' and often required 'super user' intervention. Occasionally this is still the case now, because almost no software vendor (commercial or open-source) digitally signs all their files capable of executing instructions (either natively or via an interpreter). That said, the third-party app control package stops everything it doesn't recognise or has an exception for. I don't know how far it embeds itself into the system though.

It seems like vendors like CrowdStrike swoop in on the 'easy to be compliant' gimmick.

One wonders how many are aware of Microsoft's own products in this space? App Locker comes to mind.

>One wonders how many are aware of Microsoft's own products in this space? App Locker comes to mind.

Defender is direct Microsoft product offering in this space.

This completely ignores security incident response entirely. I don’t think removing the tools that slow or prevent a hack is the answer. I think it is unfair to call it crap if you have not dealt with security incidents.

Of all the lessons that could be learned from this, I think this not the best one. We don’t call for everyone to go back to having their own datacenter because AWS had a massive outage.

> We don’t call for everyone to go back to having their own datacenter because AWS had a massive outage.

We don't? Okay, well, probably not everyone.

> I don’t think removing the tools that slow or prevent a hack is the answer.

No one ever said that. It's about removing tools that introduce a huge security risk for abysmal gain. Antiviruses are there for compliance and are detrimental to actual security as explained by GP.

CrowdStrike isn't just an anti-virus/malware solution. It also provides observability into the running processes. It integrates into threat intelligence so you can take hashes of known bad processes and see if it exists across your fleet of devices.

It can isolate devices from a network and only allow the SOC access to investigate.

If you run a hospital and find yourself unable to treat patients during an AWS outage, that should probably be a wake up call.

The difference between a vulnerability and a bug is slim. This is why I don't immediately install every single software update; official code is more likely to cut me off from my livelihood than an actual exploit taking place.

I think that Russians are now in a unique position, having learned this lesson almost universally due to sanctions. Any foreign vendor of self-updating software can, in theory, be ordered to kill your system remotely or to deliberately introduce a backdoor.

P.S. I do realize that the same applies to domestic products, especially to the ones mandatory to install, like Russian software that is required to be present on new smartphones sold in Russia. But my point is about people learning a lesson.

You completely missed the point in the article.

The correct lesson is to not use what everybody else uses. That way it won't fail the same way, at the same time.

What a ridiculous take. I'd love to see any one of the 70 Fortune 100 companies that are Crowdstrike customers turn off Crowdstrike for an extended period of time and see how long it'll take for someone to brick their entire infrastrucure with ransomware.

We live in an immensely networked world built on top of swiss cheese infrastructure (even the latest Windows is riddled with points of attack surface), unless you plan rehaul the underlying infrastructure it's simply not an option to run large scale infra without a protective layer on top.

SpaceX (private) and Tesla (#40) dropped CrowdStrike.

I immensely doubt that actually happened (saw the Musk tweet) - from working with some companies of similar size, EDR rehauls are multi-month projects. Yes they mightve dumped them as a vendor but I'm 100% certain they're not going without an EDR throughout this time.

If they are it'll be a fantastic litmus test of my claim.

They probably bought another similar solution like S1 or Defender. I don’t think they are running without a security solution which was this thread’s discussion point.

Do we have available statistics as to how many actual attacks are thwarted by Crowdstrike or similar products (and not by other mitigation strategies -- i.e. all mitigations failed, CS succeeded)?

For some definition of "other mitigation strategies", yes. The differences between CrowdStrike's EDR and any of the other top 5-10 products in its category are negligible, for example. But having strong GPO controls and a firewall and an email gateway doesn't absolve you of the need for something on endpoints. "Malware can hide, but it must run", as the old SANS saying goes, and the endpoint is where that happens.

This is an outdated philosophy that should have died long ago, and CrowdStrike's marketing itself agrees. The solution is to have zero-trust endpoints. That's what CrowdStrike claims to be doing, and that's also what it can't do. Having true zero trust endpoints absolves you of the need for something, anything, on the endpoints. It's not even as disruptive as it seems to implement.

Even if, say, banks and insurance companies could move a couple of hundred thousand Windows endpoints to fully functional zero trust tomorrow, they'd still need something on the endpoint, if only for log aggregation and detection. It is certainly awesome if the bad guys can't move laterally but you still want to know they're there.

If you want to harden the endpoints that's fine. But it's not worth doing that if it means that you now have to trust your endpoints. If the choice is between a zero trust architecture or detection, you should take zero trust every time. That said, you don't need to trust the endpoint to harden it.

Why does my log aggregation and monitoring tool need to be a kernel driver? Why can't I just use Windows Defender and a network scanner for detection, and netboot the endpoints to make bootkits useless? That would provide just as much in the way of detection, and more in the way of protection, and I still don't need to trust my endpoint, nor any third party.

From a security perspective what you're describing is a nice-to-have, and to get it you'd decuple your attack surface by adding a networked, unfirewallable, impossible to remove rootkit-shaped target for every hacker in the world to instantly own your entire network.

Banks and insurance companies already have a lot of endpoints with no kernel level EDR, those are their Android and iOS devices. It works just fine and has for a long time.

> a networked, unfirewallable, impossible to remove rootkit-shaped target for every hacker in the world to instantly own your entire network.

This old chestnut?

1. Exploiting endpoint agents is mildly popular on blogs and in conference talks. It could happen in the wild, but that's costly R&D for something that would be found and patched immediately. The economics are against it, so in practice it doesn't happen.

2. Exploits in general are actually pretty rare in the corporate world, and when we do see them they're mostly command injections like Log4Shell. This is a 180-degree turn from a decade ago, when Internet Explorer was the dominant browser, the web ran on Flash, people read PDFs in Acrobat, and Microsoft could barely patch the Office equation editor because they'd lost the source. That world is gone.

3. In the 2020s, almost every attack you'll have heard of was because of weak passwords or missing MFA. SolarWinds, Colonial Pipeline, Uber, Okta, Microsoft, UnitedHealth, on and on. Plenty of malware, zero exploits.

So to sum up, you have real risk from bad things that actually happen, and then you have tools that effectively manage that risk, and then you have hypothetical risk that never happens associated with those tools. And your position is that the hypothetical risk that never happens is so great that people shouldn't use these tools to stop the real risk that actually does happen.

Which, sure. That's a take.

> 1. Exploiting endpoint agents is mildly popular on blogs and in conference talks. It could happen in the wild, but that's costly R&D for something that would be found and patched immediately. The economics are against it, so in practice it doesn't happen.

You can't know that with anywhere near this level of confidence. For most organizations the only thing that would detect such an attack to begin with is the ERR itseld. An exploited endpoint agent in a world where endpoint agents are the cornerstone of security means that there is no guarantee it would be found immediately, especially if it's used for a targeted attack and cleaned up afterwards. Even if it was, it would be impossible to patch automatically and would require reinstalling the OS.

> 3. In the 2020s, almost every attack you'll have heard of was because of weak passwords or missing MFA. SolarWinds, Colonial Pipeline, Uber, Okta, Microsoft, UnitedHealth, on and on. Plenty of malware, zero exploits.

There were plenty of attacks that relied on exploits in the 2020s. The KEV database itself lists hundreds every year. Multiple ransomware groups relied on RCE vulnerabilities. Sometimes weak credentials are also implicated, that doesn't mean they are the only reason, and certainly doesn't mean "zero exploits".

> So to sum up, you have real risk from bad things that actually happen, and then you have tools that effectively manage that risk, and then you have hypothetical risk that never happens associated with those tools. And your position is that the hypothetical risk that never happens is so great that people shouldn't use these tools to stop the real risk that actually does happen.

The real risk that gets exploited every day is that we trust endpoints we shouldn't and don't need to. These tools do not manage this risk effectively, and in fact they make it worse because they are extremely difficult to administer when you don't integrate the endpoints in your network, and in addition to that they require you to trust one more party. The "hypothetical" risk of trusting your endpoint and integrating it in your network literally proves itself dozens of times every day.

Besides, SolarWinds and Okta showed that trusting additional security vendors is a real risk. What if instead of SolarWinds being the supply chain attack, it was CrowdStrike, and that patching the malware would require manual intervention? Supply chain attacks are hardly a hypothetical risk.

And before you say that endpoint trust and EDR is unrelated, try managing an EDR deployment without Active Directory. It's basically impossible. EDR tools assume your endpoint is trusted by default because they're too invasive to be managed otherwise.

> Why can't I just use Windows Defender and a network scanner for detection, and netboot the endpoints to make bootkits useless?

Defender for Endpoint runs in kernel mode. If you just mean plain jane Defender, well, there's a reason Microsoft sells Defender for Endpoint.

I'm not sure what a network scanner is intended to do here but if you mean a network intrusion detection system, those still exist but the market has kind of been strangled by EDR because you get higher fidelity detections and need one fewer skillset.

> It works just fine and has for a long time.

Sort of. There are a lot of controls (MDM and sandboxing and such) that go into that. I'd guess most orgs would just slap EDR on the phones if they could. Again, same result with fewer skillsets and products if so. You need a pretty severe risk to outweigh that and I'm not sure a one-time incident and the slight risk of an actual hostile compromise of an EDR vendor get there. Maybe if there are more like this.

Plain Jane Defender also has a kernel component. The point is to not have to trust any additional vendor. You can't avoid trusting your OS vendor with writing and updating kernel code but you can avoid trusting CrowdStrike.

> Sort of. There are a lot of controls (MDM and sandboxing and such) that go into that. I'd guess most orgs would just slap EDR on the phones if they could. Again, same result with fewer skillsets and products if so. You need a pretty severe risk to outweigh that and I'm not sure a one-time incident and the slight risk of an actual hostile compromise of an EDR vendor get there. Maybe if there are more like this.

There's nothing wrong with MDM and group policy in and of itself.

If orgs could slap EDRs into their mobile devices, the mobile devices would have to become less secure to begin with. The EDR seems to be a marginal risk, but the problem is in the architecture change that's needed to accomodate such an invasive product as an EDR that is a huge risk.

Windows has Defender built in and most of those companies are presumably already using an SCCM like Software Center. If they upgrade from ancient versions of Software Center (because let's be honest if they are using CrowdStrike they are probably behind the times) to more recent versions of Intune it includes most of the exact same "fleet configuration" and "fleet anti-virus" tools (including ransomware prevention/quarantine) that CrowdStrike ever provided, only this time with stuff the Fortune 100 companies are likely already paying Microsoft for. They are trading one vendor for another, but the new vendor actually owns the Operating System and has a lot more reasons not to break it with bad update deployments.

It's not like Microsoft has spent the last few decades entirely ignoring these large scale infrastructure needs, they've just been competing with these bloatware vendors like CrowdStrike with significant marketing arms and incredible sunk cost inertia. (Given most of these companies already pay for things like Intune, it's a weirdly doubled sunk cost inertia, but still a sunk cost inertia.)

What does any of SCCM and Intune has to do with Crowdstrike? Intune is already outdated btw.

CrowdStrike included some basic SCCM/fleet management "compliance checkboxes". That is one of the reasons companies were paying for CrowdStrike is "fleet management" of AV.

> Intune is already outdated btw.

I do know that part of Microsoft loves to change brand names every quarter for fun and the "parent" organization has changed a few times from different parts of Office and Windows Server to now "Microsoft 365", but as far as I can tell (and I don't keep up with it) Intune is still the active brand name: https://en.wikipedia.org/wiki/Microsoft_Intune

It's because of the merger into Microsoft 365 that I feel confident that most big Enterprises are already paying for Intune even if they aren't using the most up-to-date version and don't know what compliance "checkboxes" it solves versus something like CrowdStrike.

The Microsoft competitor to CrowdStrike is Microsoft Defender XDR (specifically Defender for Endpoint), an enterprise product that is pretty good (in my opinion) but not one that Microsoft just throws in as a sweetener with Office 365 or whatever. You will pay specifically for it. It also runs in the kernel, as most effective EDR does, so until eBPF for Windows is done you are taking the same chance with it as with CrowdStrike. It's just a matter of which company you trust more not to BSOD you.

One of those two companies also makes the OS, the other one piggybacks on it. So it seems updates to Defender XDR would be less likely to brick the OS like this.

I've never worked for Microsoft so I'm speculating, but I doubt their org structure supports that. The Windows team isn't going to be the same as the EDR team.

The EDR team might have more info about Windows internals than CrowdStrike does, but then again, maybe not; CrowdStrike already has to work pretty closely with Microsoft to get into the kernel in the first place, and Microsoft is (sometimes) pretty careful about 'bundling' antitrust concerns.

I think the feeling is that Microsoft as the company that has to field the blame for any problems in Windows has more incentive structures in place internally to "never further tarnish the Windows brand". CrowdStrike doesn't care if people blame Windows (and partially benefited in some parts of this news cycle from people blaming Windows for being the messenger), but Microsoft has nothing but skin in that game, and even if the team was in a terribly remote part of the org structure from the Windows team would still be incentivized to avoid making Windows (any more) the bad brand even if for no other reason than "shareholder value" and the equity stakes in their total compensation. (But Microsoft will have other internal incentives in place, especially the closer the team gets to Windows.)

> until eBPF for Windows is done

Red Hat says CrowdStrike themselves have sent out eBPF files that have caused all sorts of kernel panics. Any sufficiently advanced kernel scripting language is still kernel-level code. I'm not sure eBPF is exactly the savior people are hoping for, though I do still think it remains something of a general improvement if Microsoft can actually get AV vendors and others to use eBPF rather than bespoke kernel drivers.

The price is no joke. Microsoft has a lot of really good security products locked behind a paywall that security vendors know they can beat. We were looking the price for sentinel the other day and holy shit it’s costly.

There should be a law that you can’t offer security at a premium. It’s too important.

Paying less to reduce security is a baffling business model.

Maybe stop running critical infra on top of Windows? lol. Microsoft is claiming they have zero culpability here but they created this ecosystem and Crowdstrike is just playing around in it.

Ancient IT mindset says "me need domain controller. me buy win server." but it does not need to be like this.

Most companies have 3rd party software they buy which is critical to their and most of that software was designed to run on Windows. Since people are locked into this software, the vendors of said software have little incentive to switch off Windows Server. Exchange Server is prime example of this if everyone hadn't switch to 365.

This is the state of modern Enterprises.

> I'd love to see any one of the 70 Fortune 100 companies that are Crowdstrike customers turn off Crowdstrike for an extended period of time and see how long it'll take for someone to brick their entire infrastrucure with ransomware.

I've got two objections to this conventional wisdom:

1) Endpoint protection in particular isn't essential to good infrastructure. It's only a bandaid over bad infrastructure. Perhaps a moot point, given that bad infrastructure is prevalent, but I digress.

2) Every IT director asserting this should be challenged to prove it. Prove that you stopped credible threats equal to X dollars of damage or lost revenue compared to the Y dollars of actual lost revenue due to production outages. And I'm not talking about the inflated threat totals that such systems produce to justify their existance, but actual threats that absent CrowdStrike would have penetrated other defenses. Until this can be done, it's simply a hypothetical vs the actual damages inflicted by the outage.

For your second point: That line of reasoning has traditionally resulted in underfunding of security.

It’s also quite impossible in practice.

Say that you know for a fact that CrowdStrike prevented a ransomware attack. How can you accurately assess the money saved by prevention if you don’t know what the ransom would have been set at? How can you accurately assess the value of the data loss from not paying the ransom?

You can make educated guesses in both cases, but they are still strictly in the realm of the hypothetical.

>For your second point: That line of reasoning has traditionally resulted in underfunding of security.

Underfunding relative to what? Another example I'd cite would be the TSA. How much do we pay it now vs how much would we lose based on what they've truly stopped?

>Say that you know for a fact that CrowdStrike prevented a ransomware attack. How can you accurately assess the money saved by prevention if you don’t know what the ransom would have been set at? How can you accurately assess the value of the data loss from not paying the ransom?

You could estimate at an order of magnitude based on the level of access of the user and a model of your infrastructure. For example, if only one machine is compromised, you could estimate order of magnitude based on the value of the data lost, the floor being set at the days of work lost * hourly rate of the user.

Underfunding in the sense that extremely preventable real world hacks and breaches happened, with loss of customer private data. And underfunding because those companies had documented IT asking to be allowed to do exactly things that would prevent the attack.

If you want proof that real world attack would be stopped by measure, then you will leave the system vulnerable to everything your internal people were not able to exploit.

Agree, but in addition, the situation that created this problem will soon be resolved when eBPF for Windows goes to prod. It won't be possible to bluescreen a Windows box just because EDR has a bad update.

The situation(s) that make EDR valuable, however, are not going away. Not just for prevention, but for detection and threat hunt, EDR in general and CrowdStrike in particular have pretty much eaten the market that was once shared by network detection, etc.

EBPF is a good direction although I'm not sure what the maturity of the Windows fork is right now, and how many kernel apis are enabled in it - i.e. how far away are we from having enough capability to run an EDR in it

Reply on news.ycombinator.com