Hacker News Reader

Comment by siscia | Hacker News Reader

The thing I don't understand about all of this is another, much less technical and much more important.

Why the blas radius was so huge?

I have deployed much less important services much more slowly with automatic monitoring and rollback in place.

You first deploy to beta, where you don't get customers traffic, if everything goes right to a small part of your fleet, and slowly increase the percentage of hosts that receives the updates.

This would have stopped the issue immediately, and I somehow I thought it was common practices...

View on HN

It wasn't software update. It was signature database update. It's supposed to roll out as fast as possible. When you learn about new virus, it's already in the wild, so every minute counts. You don't want to delay update for a day just to find out that your servers were breached 20 hours ago.

We can see clearly now that this is a stupid approach. Viruses don't move that fast.

This situation is akin to the immune system overreacting and melting the patient in response to a papercut. This sometimes happens, but it's considered a serious medical condition, and I believe the treatment is to nuke someone's immune system entirely with hard radiation, and reinstall a less aggressive copy. Take from that analogy what you want.

> Viruses don't move that fast

Yes they do? And it’s more akin to a shared immune system than a single organism.

In this case, it’s not like viruses move fast relative to the total population of machines, but within the population of machines being targeted they do move fast.

Still, better to let them spread a bit and deal with the localized damage than risk nuking everything. There is such a thing as treatment that's very effective, but not used because of a low probability risk of terminal damage.

Cite one virus thay crashed the supposed 10 or 100 million machines in 70 minutes.

Just one.

https://www.caida.org/catalog/papers/2003_sapphire/

[SQL] Slammer spread incredibly quickly, even though the vulnerability was patched in the prior year.

> As it began spreading throughout the Internet, it doubled in size every 8.5 seconds. It infected more than 90 percent of vulnerable hosts within 10 minutes.

Worms are not technically viruses, but they can have similar impacts/perform similar tasks on an infected host.

You are off by several orders of magnitude

Also keep in mind 8.5 million is likely the count of machines fully impacted and are not counting the machines impacted but were able to be automatically recovered.

> You are off by several orders of magnitude

Can you cite something? This is HN, not reddit.

> Also keep in mind 8.5 million is likely the count of machines fully impacted and are not counting the machines impacted but were able to be automatically recovered.

Do you have evidence of this? Please bring sources with you.

Can you explain why you find this idea of fast moving viruses so improbable? Just from the way the internet works, I wouldn’t be surprised if every reachable host could be infected in a few hours if the virus can infect a machine in a short time (a few seconds) and would then begin infecting other machines. Why is that so hard to imagine?

Proper firewalling for one. "Every reachable host" should be a fairly small set, ideally an empty set, when you're on the outside looking in.

And operating systems aren't that bad anymore. You don't have services out of the box opening ports on all the interfaces, no firewalls, accepting connections from everywhere, and using well-known default (or no) credentials.

Even stuff like the recent OpenSSH bug that is remotely exploitable and grants root access wasn't anything close to this kind of disaster because (a) most computers are not running SSH servers on the public internet (b) the exploit is rather difficult to actually execute. Eventually it might not be, but that gives people a bit of breathing space to react.

Most cyberattacks use old, unpatched vulnerabilites against unprotected systems combined with social engineering to get the payload past the network boundary. If you are within a pretty broad window of "up to date" on your OS and antivirus updates, you are pretty safe.

The focus seems to have been the time limit though. All the reasons you mention are just that there aren’t even that many targets.

Microsoft puts the count at 8.5 million computers. So, percentage wise, the MyDoom virus in 2004 infected a far greater % of computers in a month: which in the context of internet penetration, availability and speeds (40kb/s average, 450kb/s fastest) in 2004 was about as fast as it could have. So it might as well have been 70 minutes, given downloading a 50mb file on dial up would take way longer than 70 mins.

To the smart people below:

It’s clear to everyone that 70 minutes is not 1 month. The point is that it’s not a fair comparison: it would simply not have been possible to infect that many computers in 70 minutes: the internet infrastructure just wasn’t there.

It’s like saying “the Spanish flu didn’t do that much damage because there where less people on the planet” - it’s a meaningless absolute comparison, whereas the relative comparison is what matters.

There's also orders of magnitudes more machines today than 20 years ago -- so it should be easier to infect more machines now than before, and yet no one can sight a virus that was as quickly moving and damaging as what crowdstrike did through gross negligence.

Be better.

This entire thread is stupid.

Computer security as a whole has improved, whilst the complexity of interconnected systems has exponentially increased.

This has made the barrier to entry for malware higher, and so means we no longer have the same historic examples of large scale worms targeting consumer machines that we used to.

At the same time the financial rewards for finding and exploiting a vulnerability within an organisations complex stack have greatly increased. The rewards are coupled to the time it takes to execute on the vulnerability.

This leads to what we have today: localised, and often specialised attacks against valuable targets that are executed as fast as possible in order to minimise the chance a target has to respond or the vulnerability they are exploiting to be burned.

Of course the “smart people belw” must know this, so it’s unclear why they are pretending to be dumb.

> This leads to what we have today: localised, and often specialised attacks against valuable targets that are executed as fast as possible in order to minimise the chance a target has to respond or the vulnerability they are exploiting to be burned.

Yup, exactly that.

So what I'm saying it, it's beyond idiotic to combat this with a kernel-level backdoor managed by one entity and deployed across half the Internet. If anyone manages to breach that, they have a way to make their attack much simpler and much less localized (though they're unlikely to be prepared to capitalize on that). A fuckup on the defense side, on the other hand, can kill everything everywhere all at once. Which is what just happened.

It's a "cure" for disease that happens to both boost the potency of the disease, and, once in blue moon, randomly kills the patient for no reason.

But now you run into the tragedy of the commons.

The fact is that this does help organisations. Definitely not all of the orgs that buy Crowdstrike, but rapid defence against evolving threats is a valuable thing for companies.

So, individually it’s good for a company. But as a whole, and as currently implemented, it’s not good for everyone.

However that doesn’t matter. Because individually it’s a benefit.

That's right.

Which is why I'm hoping that this incident will make both security professionals and regulators reconsider the idea of endpoint security as it's currently done, and that there will be some cultural and regulatory pushback. Maybe this will incentivize people to come up with other ideas on how to secure systems and companies, that don't look like a police state on steroids.

But you’re conflating a few different things here. The regulations don’t say “you must use a fragile kernel module that runs the risk of boot-locking” do they?

The underlying fault in this drama is Microsoft - third party code shouldn’t be able to have the impact it did, regardless of how it is loaded or what it does. Their commitment to supporting legacy interfaces has shot them in the foot here.

If HP pushed a dodgy printer driver (and if those still lived in the kernel) that nuked tens of millions of machines, would you be out here saying “regulators and security professionals need to re-consider printers”?

Microsoft will shit bricks, start to do something to isolate kernel modules, Crowdstrike will be the first shining user of this, and life will go on.

The malware doesn't need to infect 100 million machines.

It just needs to infect 200k devices to get to the pot: hundred million dollars of ransomware.

It's a trivial cost to pay if the alternative is CrowdStrike inflicting billions of dollars of damage and loss of life across several countries.

(I expect this to tally up to double-digit billions and thousands of lives lost directly to the outages when the dust settles.)

Trivial cost to pay from which side?

The organization like MGM and London Drugs?

ILOVEYOU is a pretty decent contender, although the Internet was smaller back then and it didn't "crash" computers, it did different damage. Computer viruses and worms can spread extremely quickly.

> infected millions of Windows computers worldwide within a few hours of its release

See: https://en.wikipedia.org/wiki/Timeline_of_computer_viruses_a...

No they're in the wrong. They didn't test adequately, regardless of their motive for not doing so. Obviously reality is not backing up your theory there

FYI, both the following statements can be true:

1. Crowdstrike didn’t test adequately

2. Viruses can move pretty fast once a foothold is gained

https://en.wikipedia.org/wiki/SQL_Slammer

There is no real "speed limit" on malware spread.

No, but there are impenetrable barriers. 0days in paricular are usually very specific and affect few systems directly but even the broader ones aren't usually followed by a blanket attack that pwns everything and steals all the data or monies. Just about the only way to achieve this kind of blast radius is to have a kernel-level backdoor installed in every other computer on the planet - which is exactly what those endpoint "security" systems are.

It’s quite impressive really — crowdstrike were deploying a content update to all of their servers to warn them of the “nothing but nulls, anti-crowdstrike virus”

Their precognitive intelligence suggested that a world wide attack was only moments away. The same precognitive system showed that the virus was so totally incapacitating that the only safe response was to incapacitate the server.

Knowing that the virus was capable of taking down every crowdstrike server, they didn’t waste time trying it on a subset of servers.

When you know you know.

Surely there is a happy medium between zero (nil,none,nada,zilch) staging and 24 hours of rolling updates? A single 30 second or so VM test would have revealed this issues.

There should have been a test catching the error before rollout, however this doesn’t require a staged rollout as suggested by the GP comment, testing the update at some customers (which would still be hosed in that case), it only requires executing the test before the rollout.

But why does a signature database update have to mess with the kernel in any kind of way? Shouldn't such a database stay in the user land?

The scanner is a Ring 0[0] program. Windows only has 2 options 0 and 3. 3 won't work for any kind of security scanners, so they're forced to use 0.

The proper place would be Ring 1, which doesn't exist on Windows.

And being a kernel-level operation, it has the capability to crash the whole system before the actual OS has any chance to intervene.

[0] https://en.wikipedia.org/wiki/Protection_ring

Why is so?

All modern OSes only use ring 0 and 3. Intel is considering removing rings 1 and 2 in a future revision for that reason: https://www.intel.com/content/www/us/en/developer/articles/t...

Historical reasons. Windows NT was designed to support architectures with only two privilege rings.

That's a question for Microsoft OS architects

Because kernel needs to parse the data in some way and that parser apparently was broken enough. Whether it could be done in a more resilient manner, I don't know, you need to remember that antivirus works in hostile environment and can't necessarily trust userspace, so probably they need to verify signatures and parse payload in the kernel space.

Yup. If they were delaying update to half of their customers for 24 hours, and in that 24 hours some of their customers got hacked by a zero day, say leading to ransomeware, the comment threads would be demanding their head for that!

Even if it is a staged rollout why would one do it in 24 hour phases ? It can be a hourly (say) staggered rollout too.

Sure. And if someone showed up here with a story about how they got attacked and ransomwared enterprise-wide in the however many several hours that they were waiting for their turn to rollout, what do you think HN response would be?

Hmm, maybe you could have companies pay more to be in the first rollout group? That'd go over well too.

True, there will be comments blaming CS for not doing faster rollout. But there would be some comments empathizing with CS viewpoint and pointing out the conflicting compromise between velocity, and correctness. Even now I think the comments wouldn't have been unequivocally critical, of CS, if the hosts affected were a variant of windows (say issue was seen on version of windows 10 which was two update behind),there would have been some emphasizing the thorniness of the problem and sympathetic of CS.

It doesn't matter what kind of update it was: signature, content,etc. Only thing that matters is does the update has a potential to disrupt the user's normal activity (leave alone bricking the host), if yes ensure it either works or have a staged rollout with a remediation plan.

You do want to fuzz test it like crazy. Can be automated. Takes minutes, saves billions

Thanks for the clarification, this makes more sense.

Even if there was a canary release process for code updates, the config updates seem to have been on a separate channel.

The expectation being that people want up-to-date virus detection rules constantly even if they don't want potentially breaking changes.

The missed edge case being an untested config that breaks existing code.

Source: Pure speculation, don't quote this in news articles.

Considering the impact this incident had they definitely should have a large staging environment of windows clients to deploy first.

There are so many ways to avoid this issue, or at least minimize the risk of it happening, but as always profits come before people.

They don't seem to dogfood their own software. They don't seem to think it's very useful software in their own org, I guess.

It's answered in the post (in the thread) as well. But for comparison, when I worked for an AV vendor we pushed maybe 4 updates a day to a much bigger customer base (if the numbers reported by MS are true).

I'm curious, what did your deployment plan look like? Phased/staggered, if so how?

It was a long time ago and I wasn't as involved with this, so I don't know with certainty what was used and how. We had multiple channels for major product versions + beta customers on the most recent one. On top of these we could stage different CDN regions. There were different types of data you could update and those might have been treated differently (e.g. simple file signatures vs definitions for heuristics).

"Blast radius" seems... apt.

It would be rather easier to understand and explain if it were intentional. Likely not able to be discussed though.

Anyone able to do that here?

Reply on news.ycombinator.com