Hacker News Reader

Comment by addicted | Hacker News Reader

The only reason this could be funny is because the software industry has found a way to excuse itself from any liability.

There is no other industry where someone could cause so much damage and laugh about it. Least of all because the liability itself would have led to its collapse.

Can you imagine a company hired to reinforce a bridge to protect it from damage from a ship instead causes its collapse.

How long is that company gonna last? Even if no one dies or is injured it will be run out of business.

Only in Tech can such a company not only survive but laugh about it.

And that’s even before we get to how amateurish the mistake these guys made was.

View on HN

Fuck up the world's computers, piss off all of its IT teams, and then send people $10 uber eats gift cards as if that'll get you anything, maybe a coke at best.. but its further admitting fault. That's like, a tip. Like here, have a sandwich while you fix our fuckup.

They don't care. They'll all get 6 figure bonuses too in management for 'weathering the storm' after the mishap and probably get more money because look what they can withstand, literally technology-murder and get away with it.

It's almost movie-grade evil villainy tier stuff lol

Because people on the receiving end are the same - they accepted and rolled out the update without even as much as “canarying” it. SolarWind was the same - the customers weren’t bothered even by mismatched integrity hashes. It is a tacit pact in our industry - we all screw up and cut the slack to each other. Who will cast the first stone?

Crowdstrike can force-push an update at any time of their choosing that the connected device will grab and load, is my understanding.

Don’t you see that you’re only enforcing my point?

No, because "canary" in the context that you used it, has a specific meaning. If you believe they should have tested CrowdStrike more or been more skeptical of their claims before licensing, that's independent of the user/administrators doing canary-style testing.

What’s the default? And what did their technical account manager recommend? My guess, no canary ring.

> they accepted and rolled out the update without even as much as “canarying” it.

Well, no; AIUI part of the problem was precisely that this update was pushed in such a way that it skipped any canary system in place. There might be a separate conversation to question what percentage of their users were taking advantage of its staged rollout features, but it's rather immaterial when the incident in question bypassed them even if users had configured it sensibly.

But the customer installed CS software could do this. So they are partly to blame. I do not think you will find that tesla would allow a third party update to its car. Or a oil rig would allow third party updates to critical parts of its systems. So its understanding the context. I think a lot of places this is an risk that is ok. But maybe not everywhere. And I hope some companies with critical systems will learn from this

> But the customer installed CS software could do this. So they are partly to blame.

It depends on if/how it was communicated. If there's a big red box in the user manual that says, "this software might take updates that completely bypass any phased rollout you configure", then yes it was probably irresponsible to use it. If, however, the software lets you configure phased rollouts and fails to mention that they might just get ignored, then I don't see how the customer can be blamed at all. (And in both cases, if CS shipped such an update with exactly zero testing whatsoever, which strains credulity but is what I've read, then they still get most of the blame.)

Honest question, what should they do? Uninstall the company and give up? Ignoring their actual response, what is a good response to this?

If my company causes billions in damages and endangers human lives, I can't imagine why my company isn't bankrupted and dissolved.

Ok, but the market makes that decision, not the company. Crowdstrike has no choice but to accept the sentence the market hands it. It’s just that the market appears to have sentenced it to…barely anything. Blame those still using CrowdStrike after this incident.

If your company causes damage at society-scale (hell, even if it does major damage to one person's life), the state should be ready to intervene and make the company pay the tab for the damage they caused? Like, that doesn't sound very controversial.

Yea. Their contracts likely have clauses for all of that. I say likely, but we already know this is true because it's come out.

The thing is, crowdstrike isn't the only incompetent party here. Many major companies (looking at Delta) probably made it worse for themselves with a very poor response after.

So should crowdstrike pay beyond a reasonable measure because of Delta's poor response?

No contract clause can protect you from a gross negligence tort.

(Or equivalent in one's respective civil law system.)

This might be the easiest gross negligence tort case to show and litigate-- still hard but if everyone starts the lawsuits they can not pull the contract to protect them. They will try of course and they will fail in most but the obvious cases.

What you can not sue them for is not forseeable damages -- e.g. I lost my dream job because the computer died during the interview. But ceasing operations of a company is generally fair game. And plaintiffs can argue that no reasonable person could forsee and mitigate against this disaster so the failure is not due to plaintiff's "fault" negligence.

Reckless typically requires conscious disregard of risk. Arguably, that would require Crowdstrike emails from programmers saying “this is risky, we need to test it” and management responding “F it! We’ll do it live!”

If nobody in CS realized how dangerous their process was, it’s not reckless.

That's interesting but my sniff test isn't passing. "Reckless driving" doesn't require me to know it's a bad idea to do 100 miles per hour in a 25, it is reckless whether I realize it or not right? IANAL but the only thing I can think of requiring knowing to be at fault is slander, at least in the USA.

> Blame those still using CrowdStrike after this incident.

I think you'd have to ask "why are they still required to use CrowdStrike or any AV provider?" I think once you find the answers to these questions you realize this is not a properly functioning product market.

How you can then build a publicly traded company on the back of a complete and total lie is another subject, but it's certainly also implicated in the above questions.

Actually generally the legal system would decide that, not “the market”.

I.e. investors have assigned roughly zero probability to CrowdStrike bearing the full cost of this incident, and set the market price accordingly.

The market as a tool for punishing bad players is far from perfect. It's why we still have monopolies and see consumer antitrusts and other similar legal suits in court. Advocating for shifting blame to customers still using Crowdstrike is ignorant of the problem and further signals a dishonest approach to the issue at hand.

Equifax, for example. They probably caused way more damage leaking all our info

The company should be fined so heavily that they become a goverment asset.

With current shareholders wiped out entirely, it should then be re-listed.

IMO the company that this should have happened to is PG&E. I think California could have forced them into liquidation and bought their assets. No bailout required, complete loss for shareholders, and CA could potentially have fixed much of its disastrous utility situation at a reasonable price.

A company-ending fine or judgment against Crowdstrike wouldn’t come with any great reason for a public takeover — Crowdstrike could cease to exist and the overall ecosystem would be fine.

Outside of Alaska and Wyoming, Silicon Valley has the worst power and internet of anywhere I've ever lived (worse than AR, MN, and ND by a longshot), measured either in incremental cost or uptime/availability. The fact that PGE keeps requesting additional rate increases "for fire safety" and immediately kicks those back to shareholders isn't a great look either.

Hide themselves from the face of the world for at least 5 years until people somewhat forget about them.

Crowdstrike should be held accountable but so should any reasonably sized enterprise that allows code to be pushed to their whole enterprise without testing. All the large enterprises I've worked for required Windows patches to be tested before being pushed to production why are crowdstrike updates treated differently?

It clearly states in the post mortem that rapid response updates were pushed globally without any say from the customer: https://www.crowdstrike.com/wp-content/uploads/2024/08/Chann...

So in theory if a company like crowstrike were compromised by criminals/disgruntled employee/NSA the impact could not be avoided?

Given the cybersecurity landscape it is not unusual for security software updates to be pushed globally without the option to test or even stagger them. When a potent new vulnerability becomes public knowledge (or semi-insider knowledge) and especially if there's already a PoC available, organizations only have minutes to a few hours before threat actors begin utilizing it.

APTs and organized crime groups have 24/7 staff to weaponize and integrate new vulnerabilities into their workflow as rapidly as possible, or have other contracted other groups to provide this service.

So, you would prefer that they don't accept this "award", and in so doing admit that they messed up?

And honestly, crowdstrike is more likely to go under than a company that failed to re-inforce a bridge. Their mistake caused measurable harm to many well funded companies that have the resources to sue crowdstrike in court.

If crowdstrike survives, it will be because there isn't a lot of competition in their market, not because they can excuse themselves of liability.

I would have liked to see Crowdstrikes legal councils face as they accepted this award. There is no way they ran it by them.

Statistically speaking it seems likely people really did die from this mistake, if only indirectly due to for example delays in medical care caused by the outage.

It would be hard to imagine it didn't: https://www.washingtonpost.com/business/2024/07/19/windows-o...

I don't think this is all that accurate. In the engineering space, Boeing has so far accepted responsibility for two fatal crashes and the fucking door falling out of an airplane and is still in business.

We've professionalized industries like engineering and medicine because incompetent practitioners are a threat to public health and safety. Software is now in everything and incompetent practitioners have been a threat to public health and safety for a long time now yet we do nothing about it.

Found the gatekeeper

Make sure to pick a surgeon from the street next time you’ll need an operation. Don’t want to gatekeep the profession, after all.

Not necessarily from the street, but I do expect bio graduates to be trained in health care at scale, without being limited by residency programs. There's no point training 10 biology graduates only for 9 of them to work as waiters or on OnlyFans.

Crowdstrike and countless other software failures before it is literally the proof that the gates need to be kept. The only question is whether we start doing our own gatekeeping or it eventually gets forced on us by heavy handed legislation like it was for doctors and civil engineers.

I'm sorry, when did the switch flip in this industry where we decided we didn't want to hire people with expertise and experience?

In construction, Grenfell happened and witnesses demanded immunity from prosecution to testify, because they knew they’d broken numerous laws in its construction and certification. Residents at similar buildings are the ones paying to make them safe for habitation, not the crooks that built them. Professionalisation is not a magic bullet.

Blaming individuals when systems fail: does it work reliably?

And how can certification work across borders between jurisdictions?

I've seen a fair share of engineering disasters in my own developed country where a few signatures by engineers didn't prevent the causes.

Regulations and jail doesn't seem to be enough of a disincentive? How do you force someone to do a good job?

Open source is not very compatible with certification.

Certification either (1) makes all software proprietary or (2) requires people to sign that particular open source software is safe or (3) maybe we should disallow the liability evasion clauses of open source software licenses?

As if "professionalized industries" like engineering and medicine don't ever commit mistakes.

The time to take yourself seriously is before the stuff happens. Being stuffy now doesn't add anything. Accepting the award means they have something to show every single new hire and everyone in that office will have a physical reminder to do better in the future.

I get the other points about consequences, but I don't think accepting this award is in anyway problematic. It's one of those things that I expect only people that care about "appearances" would complain about.

I agree. They show up, cop to it, and collect a memento mori that will hopefully help motivate improvement in the future. They have a lot of work to do to repair their reputation, and I don’t think they’re foolish enough to think that this is anything more than a small step on a long path.

Ever had someone make a mistake that cost you time or money and then tried to laugh it off as no big deal? That's what this feels like.

The security industry needs to grow up.

The best thing they should have done is fire the CEO, apologized profusely, and then use their army of sales people to help make things right on a one on one basis with their customers.

A less charitable way to look at it is that they weren't taking things seriously before the incident, and they're still not taking things seriously now.

What the most appropriate way to view it is, I don't know. I think I'd need to know way more than I do about Crowdstrike leadership.

This wasn't their first serious blunder this year even, just the most damaging and visible. The nature of their mistakes seem to be exceedingly preventable too, with them failing at textbook SRE practices. Their CEO has now been at the helm of two different companies that have had similar problems under his leadership. The evidence keeps piling up and people want to keep making excuses for negligent behaviors. Why should we excuse facts for hypotheticals?

I'm sorry, how has CrowdStrike at all demonstrated that they're going to do better?

Wait till you learn about finance, or oil and gas, or mining, or countless other legacy industries that quietly run America since the Rockefeller days. Many parts of Texas lost power for weeks due to Beryl and all the power companies got was a slap on the wrist (despite being explicitly warned about this scenario many times).

Hell look at Grenfell in London! 72 people died and the people that designed it wanted immunity from prosecution to testify at the inquiry! Similar buildings are charging the cost of upgrading the cladding to residents instead of doing the right thing and eating the cost themselves.

Companies are too easy a target...

What were the regulatory incentives? Was the electricity market designed to encourage resiliency?

There was a systematic fault - it needed a collective solution not one that relies on individual companies doing the right thing???

You would not want for a sympathetic ear if you also criticized these companies. The point is not that CrowdStrike is uniquely incompetent, not at all. Every critical organization needs to be held to a higher standard, not just incompetent security firms.

I can think of at least one industry where the price of failure is almost always borne by the users and not the companies. Very closely integrated with tech as well.

100% agree. As mentioned in another post, the acceptance of this joke award is completely tone deaf.

Hospitals, banks, airlines, governments, and of course various IT operations at companies that are forced to use this endpoint security crap and Windows were impacted. Many people suffered degraded quality of care at medical facilities. Surgeons losing access to critical imaging/labs during surgery. Probably many canceled and rescheduled surgeries as well.

People’s flights were delayed or canceled. Imagine having to take a last minute flight to visit a loved one on their death bed only to get canceled because ClownStrike shit and incompetent IT departments/CTOs fucked them over.

Many people/businesses unable to access critical banking services.

Then the amount of lost productivity for office workers. Many hours lost for IT folks, often even working into the weekends. Time lost to dealing with ClownStrike bullshit when that time could have been spent with their families and friends.

Fuck ClownStrike, George Kurtz, and this latest clown, Michael Sentonas

It just illustrates how low the stakes are in tech really.

This attitude is why we have a culture of fear and do-nothings.

To add insult to the injury, these people call themselves "software engineers".

There is sparsely little real “engineering” that goes on in the field of “software engineering”, industry wide, in terms of ensuring that our software is reliable and secure. Performance and development velocity seem to take the driver seat always, in a whole lot of software.

See also this talk by Bryan Cantrill, “Scale by the Bay 2018: Rust and Other Interesting Things” https://youtu.be/2wZ1pCpJUIM where he talks about software platform values, in the sense of what different programming languages and other things focus on. He also touches on the fact that while higher level layers in the stack might value security it kind of falls apart when the very microcode in our CPUs sacrifices security for performance.

This was in a period of time where Spectre https://en.m.wikipedia.org/wiki/Spectre_(security_vulnerabil... and Meltdown https://en.m.wikipedia.org/wiki/Meltdown_(security_vulnerabi... had reared their ugly heads.

> There is sparsely little real “engineering” that goes on in the field of “software engineering”, industry wide

There surely is actual engineering but it's scattered unevenly across companies. It's funny that Crowdstrike did fuzz their code but didn't even check for correct arity. I think that the Cybersecurity industry isn't such a strong adopter of sophisticated engineering techniques as for instance in Web development where new testing techniques evolve every few years.

I really don't think that's true. All software is undertested and it's likely that there isn't a significant differences between web apps and security apps.

Having said that, writing ring 0 drivers an unsafe language sounds like an invitation to disaster. That's what went wrong with CrowdStrike. You don't need any testing to avoid crashing the OS when given a bad virus definition file. (Making the virus definition file do something useful... sure, you're gonna need tests for that.)

Performance is dead in the trunk next to a shovel and some quicklime. I know people say they take performance seriously, but as someone who is also a user of commercial software, I am reminded of James Baldwin: "I don't believe what you say, because I see what you do".

On the other hand (maybe I'm just playing devil's advocate here): nobody died (I hope at least! It's possible if some hospital equipment, 911 calls, ... failed...) from this incident despite being such huge scale that almost everyone knows about it. It's almost as if humanity can be... fine... when all this computing equipment fails.

Then what value do they provide?

Many hospitals, including emergency rooms, were shutdown. Maybe no single death can be directly tied to the event but I'm pretty this effectively resulted in greater death.

Maybe some of those inconvenience had to time stop and contemplate the world but there are parts of the world where computers stopping don't leave people just fine.

Agreed and I’m sure there are tons of anecdotesb just in this community. My daughter for example is a night shift labor and delivery nurse at a level three metro hospital. They were heavily impacted. All of the phones were down, all of their internal messaging was down, translation services are down, it was a rough couple of nights.

I don’t have any direct experience with crowdstrike, but in my experience with security vendors in particular, they make it very difficult for customers to inject useful change management into the process. I’ve been in infosec for nearly thirty years and “my people” also need to shoulder some of the blame for catastrophizing delays in delivery of updates to preventative and detective controls. I’ve always known this but spending the last year ‘outside’ in central platform delivery and operations for a large financial has really brought that to light. Fortunately I know how to speak security so it helps us navigate but many aren’t so fortunate.

Stopping air travel has probably canceled a lot of unnecessary meetings and slowed down global warming.

Maybe events like this provide value in that they indentify which systems are actually mission critical.

It also probably caused a lot of people to not be able to say goodbye to their loved ones, or miss their holiday, or whatnot. People don't travel exclusively for frivolous reasons.

I am not suggesting that airlines should see their operations as non-critical (and let a third party make arbitrary untested updates to their critical systems).

But maybe some organizations using air travel can learn from this. I am still hoping that the pandemic will have had some lasting positive effects (on top of all the pain it has caused).

Reply on news.ycombinator.com