Hacker News Reader

Comment by lqet | Hacker News Reader

Can anyone explain how CrowdStrike could possibly fix this now? If affected machines are stuck in an endless BSOD cycle, is it even possible to remotely roll out a fix? My understanding is that the machines will never come to the point where a CS update would be automatically installed. Is the only feasible option the official workaround of manually deleting system files after booting into the recovery environment? How could this possibly be done on scale in organizations with tens of thousands of machines?

View on HN

There are orgs out there right now with 50,000+ systems in a reboot loop. Each one needs to me manually configured to disable CS via safe mode so that the agent version can be updated to the fixed version. Throw bitlocker in the mix which makes this process even longer, we're talking about weeks of work to recover all systems.

CrowdStrike itself will not fix anything. They published a guide on how to workaround the problem and that's it. Most likely a lot of sales reps and VPs will be fielding calls all over the weekend explaining large customers how did they manage to screw up and how much discount will they offer on the next renewal cycle.

Legally, I think somewhere in their license it says is that they're not responsible in any way or form if their software malfunctions in any way.

Small companies get the shitty generic license.

Big companies negotiate liability terms.

> Big companies negotiate liability terms.

Never heard that in the context of the software licenses.

Pretty standard in enterprise b2b, most of the sales cycle is in contracts

Well, probably no? I've never seen liabilities in dollar value, or rather any significant value. Also I saw our company Ceowdstrike contract for 10k+ seats, no liabilities there.

This isn't a Github project with a MIT license. When you do B2B software, there aren't software licenses, there are contractual terms and conditions. The T&Cs outline any number of elements but including SLAs, financial penalties for contractual breaches, etc. Larger customers negotiate these T&Cs line by line. Smaller customers often accept the standard T&Cs.

Penalties, as far as I was involved in vendor discussions, are a part of the negotiation only when the software provider does any work on the client's premises and are liable to that extent.

For software, you don't pay penalties that it might malfunction once in a while, that's what bug-fixes are for and you get offered an SLA for that, but only for response time, not actual bug fixing. Where you do get penalties and maybe even your money back, is when the software is listed as being able to do X,Y,Z and it only does X and Z and the contract says it must do everything it said it does.

> Big companies negotiate liability terms.

I have never heard of that. Can you point to some examples?

Not SLA's (which are standard), but actual liability? E.g. if we brick your computers we'll pay for replacements and lost employee productivity?

Yep, I've been involved in many vender contracts at my company and the contracts take weeks to months to finalize because every aspect of the agreement is up for discussion. Even things like SLA's (including how they're calculated), liability limitations, indemnity, recourse in the event of system failure are all put through the ringer until both sides come to agreeable terms. This is true for big and tiny venders.

> Legally, I think somewhere in their license it says is that they're not responsible in any way or form if their software malfunctions in any way.

I really should add this to my resume and see if it’ll work.

Nah, it only works for corporations. Peons still have accountability.

I think about this all the time.

Like if I kill someone of course I go to jail. But if I get some people together, say we're a company, and then kill 100 people, nobody goes to jail. How does that work? What a huge loophole.

While it sounds funny, it doesn't work like that. We'd be having a real corporate shootouts everyday all over the place :))

Phillips (the company) basically killed people with malfunctioning CPAP machines (which are meant to help against sleep apnea) and no one went to jail. So that's a practical example.

I don't think that's true in this case. I've never heard of an individual employee who introduced a bug being legally liable for it.

It's already the norm for devs to not be responsible for software malfunctions. They can choose to end their relationship with you, but they can't sue you for damages.

"THIS SOFTWARE IS PROVIDED AS-IS..."

I think I preferred it AS-WAS.

But they've already fixed it.

"CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes."

How they've reverted changes on non-booting PCs, goodness only knows... ;)

Sounds like people in some of these environments will be doing their level best to automate an appropriate fix.

Hopefully they have IPMI and remote booting of some form available for the majority of the affected boxes/VMs, as that could likely fix a large chunk of the problem.

Imagine if North Korea comes with a statement, that they did it.. It would spawn such amount of work internally at CS to proof if it was intentional or a simple mistake.

Amazing idea

Reply on news.ycombinator.com