Microsoft says Delta's ancient IT explains long outage after CrowdStrike snafu | Hacker News Reader

Microsoft says Delta's ancient IT explains long outage after CrowdStrike snafu

arstechnica.com

46 pointsthunderbong62 comments

My wife worked at Travelport in the late 2010s. They ran a lot of Delta’s systems but Delta of course has their own infrastructure. Some of the history of Delta’s teams is intertwined with the companies that were merged together to form Travelport.

From what I could tell from her talking about work it wasn’t particularly far off the state of the art at the time. They had a fancy new AWS hire to help lead them into AWS from mainframes. That didn’t fully work out but they still had a bunch of containerized .Net and Java running.

Mostly this whole back and forth argument sounds like a ball of pain with everyone upset at each other and very little of it mattering unless lawsuits are filed. If Delta is using Windows at IBM it doesn’t make them “ancient.”

> containerized .Net and Java

Gross. If that's the alternative I would rather stick with mainframes. People forget that it's perfectly fine and even better to just run code directly in many cases.

Containers are not VMs, code runs directly on metal and performance is pretty-much-native.

https://web.archive.org/web/20150218172511/http://domino.res...

This is from 2015, and it improved from there (e.g. the daemon-less podman matured, NAT is no longer needed). That said, I'd happily take even the performance impact of VMs and order a few more servers, if it means that I won't have to deal with dotnet/java runtimes being shared by multiple applications.

If you need a container, your stack (dotnet/java) sucks. That's why Amazon deprecated the Go container, cause it's just not needed

Yes those langs have large runtimes. But even for compiled native languages, if you have a lot of dependencies you will require containers sooner or later. Go doesn't really have the huge dynamic-linking ecosystem. But, if you do and then project X and project Y diverge on versions... you have a problem.

Also containers can automate deployment easier. Like the line from bare server -> full deployment is much faster if you use containers.

The larger the org the slower they are with updates. Some global companies skip one or two versions of Windows because of the time it takes them to upgrade their systems. Microsoft is trying to deflect the blame, but it is their policy of maintaining backward compatibility that bit them. If you tell enterprise they can run their software on the latest version of your OS there is no incentive to upgrade, because they can always do it later. Steve Jobs knew how enterprise clients think from his time at NeXT and killed Apple's enterprise division when he came back. When large customers pushed him to create enterprise sales and support team he told them to go the nearest Apple Store, because he did not want them to limit his ability to change the OS.

Well, that is why Apple is meaningless on the enterprise server room, and all their attempts at it flopped.

That’s not why. It was overpriced and didn’t have a lot of software.

Linux distros make changes and they ate Microsoft’s lunch in the server room.

> Linux distros make changes and they ate Microsoft’s lunch in the server room.

This statement is a little funny considering the only reason the CrowdStrike situation was such a big deal was because of how many high-profile companies are using Windows Server.

You wouldn't want to live to see the day when the same situation happens on Linux servers.

Not only it already happened, the world hardly noticed as the systems got affected, given the lack of relevance on the affected systems.

https://www.theregister.com/2024/07/21/crowdstrike_linux_cra...

Noone noticed because most Linux servers does not run Crowdstrike Falcon. If the same number of Linux servers as Windows servers were affected, most of the Internet would have been down.

Crowdstrike is not relevant on Linux systems. don’t get it confused.

The AWS stats are very clear, Microsoft is way behind in the server world.

Crowdstrike literally disrupted Linux systems this year. Are you arguing that it isn't widely deployed enough to be relevant?

Why is that funny? They can lose 75% of their market share and still have millions of deployments.

UNIX free beer eats Microsoft licenses on the server room.

Which is quite a different point.

Naturally Apple's UNIX flavour came with Apple's tax, and not free beer one, as an additional bump, besides the software policies.

Also note that the chaos on the desktop from was caused by 1% of the Windows desktop market share.

The Year of Desktop Linux with its 2% is nowhere closer to be used in such critical infrastructure.

The price was secondary. The ecosystem for windows was just garbage for web, VM, container, etc stacks. Much like they missed the boat on mobile, they missed the boat on transitioning to cattle from pets.

Case in point, Linux ate Microsoft in the university datacenters where Microsoft was literally giving it away for free.

Hardly a case in point, UNIX has always been universities baby, it is thanks to them that it actually had a life outside Bell Labs.

Not in Eastern Europe or India where Microsoft was giving away a ton of licenses. The students rejected them anyway.

That isn't the information I have from our offshoring partners.

Your offshoring partners are ignorant. There is no equivalent partner representing open source.

You’re living in a denial zone where you think something is winning because there is no competitor making more money. The reality is people realized they didn’t need to pay at all and moved on with 10-100x the scale outside of the garden of stupidity.

How is that not a case in point? Cost is irrelevant and people chucked that shit to the side.

Also, you’re mixing up the early days of UNIX with Linux, which were essentially decades apart. I watched Linux eat the entire campuses of universities and replace windows in the early 2000s. The UNIX and BSD life support cycle you are talking about is irrelevant shit from computer science departments where windows was never welcome but the scale was constrained to the department.

That was secondary to startups using Linux, because they could not afford Microsoft and because at the time IIS was as tight as a sieve. Demand for people with knowledge of Linux drove Linux adoption at universities.

It was overpriced and didn’t have a lot of software.

That's not how I remember it at all. Apple XServers cost about the same as the competition and the XServer RAID was a bargain compared to similar hardware offerings at the time. And although I never used the software, based on the demos I saw from their sales team, it looked really nice and I never heard anybody complain about it. The aversion wasn't price or software, it was just that people didn't really trust Apple support and service.

Wtf are you talking about? If you never used any software that ran on it, you’re completely unqualified to evaluate it.

My point is that I worked at a couple of different places that where considering XServer RAID setups, and as part of that I read everything I could find, attended live presentations of the hardware and talked to people online who where using XServers. And in that process found no reason to write off the hardware of purely software grounds. If you worked extensively with the software and found it a deal breaker, then I'll defer to your experience, but at the time I didn't run into anybody with anything overly bad to say about the software.

They tried to enter that market a couple of times with MacOS and AIX, but ultimately abandoned it. They are happy making their hardware pluggable into corporate networks. It's your choice, if you want it.

Still, it flopped.

Huh? I don't see how MS has any blame in any of this. Client chooses not to upgrade systems. Client chooses to use crowdstrike as a vendor. Client chooses not to accept free support. At what point was MS supposed to do something here? There are loads of things to complain about or take issue with when it comes to MS, but I don't see how this is one of them.

Comment seems to imply it's MS fault for maintaining backward compatibility

Essentially, he is stating being good at backward compatibility is the problem :-/

Microsoft wants you to upgrade and support ends for old version for precisely that reason, to get the customers to pay (like SAP's model). I don't think MS will ever tell a customer go ahead use a unsupported older version, from both a liability perspective and wanting to make the sale perspective. If Delta wanted to cheap out, don't blame the vendor

On the other hand, Microsoft will sell you “extended security updates” for your out of support OS. Third hand information, but I was told the cost was something like $100/seat year one, $200/seat year two, etc.

Actually NeXT flubbed enterprise sales because they didn't ensure that their OS was fully POSIX-compliant. POSIX-compliant OSes didn't need special waivers in order for the Federal govt to buy them; that is why the CIA and NSA were NeXT's main customers, because no one requires spy agencies to buy COTS software.

If Jobs had spent just 2 engineer-years or less of engineering time to fix POSIX compliance, their market would have been many times larger; something HP's workstation division and Sun were very happy about, I am sure!

For NeXT, as Apple, the POSIX/UNIX story has always been the same as on Windows, a means to bring the ecosystem into the platform, not to make software there and deploy it elsewhere.

There is even a surviving video from a NeXT meeting, where Steve Jobs talks about this strategy.

I'm not sure I follow your reasoning.

If backward compatibility actually was that great and clients actually believed it, why would they wait around for ages before upgrading? It should be a no-brainer.

Instead, what I've seen in practice, is that people are very much afraid their random crap will break. They absolutely don't believe the backwards compatibility is a thing, at least not without "testing".

The company I work for has a bunch of software that the providers only "support" on versions of windows server that are at least one or two steps behind current. Hell, I've even seen this for things which seemed pretty basic (think random web app) running inside a JVM.

I don't personally use windows all that much, so don't really have first-hand experience, but my impression is that people are very much afraid of upgrades, even when running pretty "basic" things (as in "office software", not things which manipulate low-level, old interfaces or similar)

Suffer a supplier failure, and your supplier is motivated to prove you are at fault.

Microsoft didn't supply, recommend, or specify CrowdStrike.

They did supply Windows, and Microsoft is the one throwing shade.

from what ive read microsoft is partially liable for allowing crowdstrike to make updates the way they did, forcing updates and then not detecting and disabling the affected driver. maybe im wrong. seems like theyre finger pointing and that makes me think theyre guilty even more.

Windows did not disable the driver because Crowdstrike marked their driver as critical for boot. Had they not done that, Windows would disable the driver automatically.

didn't you hear? the EU forced them to let Crowdstrike to be able to do it...

This is the EU's fault :-P

The article doesn't really explain anything to an extend that would all us to conclude anything regarding the age of those systems.

The comment "because it runs on those providers' systems" is also a bit weird. Do they run on AS/400? Or is it simply because IBM would let Microsoft employees touch any IBM owned and operated infrastructure, probably the later. In my experience dealing getting access to infrastructure run by IBM (formerly SoftLayer), then yes, things might be ancient. We required VM with Windows XP to even run their VPN solution and that barely worked anyway.

I think it’s absolutely bonkers that CrowdStrike software is basically an RCE

"Security" software has always been like that. What's bonkers is the state of regulations and auditing where putting this kind of crap on your systems is somehow seen as increasing your security rather than decreasing it.

The guard watching your building needs to have full access to every room so that they can check if a bad guy got there without a key.

You need to trust the guard implicitly.

"Ancient" would be fine. At this point, we're pretty much back to a 3270 displaying data from a mainframe anyway. Use DOS and an 8514/A graphics card to access the "cloud" and we'll be better, faster and more secure than anything Javascript based.

What is NOT fine is the lack of a disaster recovery plan. Your disaster recovery plan should be able to cold start yourself within 24 to 48 hours. It's clear that Delta simply didn't have anything resembling a disaster recovery plan.

>was being serviced by other technology providers, such as IBM, because it runs on those providers' systems, and not Microsoft Windows or Azure

Another cute quote by M/S. I hope this goes to trial with a Jury. Then the truth will come out.

Crowdfare crashed only Windows Systems, not IBM Mainframes or AS/400 or AIX or Linux or BSD.

The blame is on Crowdfare and Windows only, not IBM systems.

Every PC we have has IPMI. We can re-image any PC even--if it can't boot--remotely. Even if we don't have the bitlocker key, etc.

IPMI?

https://en.m.wikipedia.org/wiki/Intelligent_Platform_Managem...

Intelligent Platform Management Interface

Most severs have a management microcontroller in them that is capable of administoring the main system, typically including (but not limited to) full keyboard video and mouse access.

Typically this is exposed through a dedicated ethernet port, allowing administrators to remotely manage their servers.

> PC .. IPMI

Since IPMI is usually on servers, did you mean Intel vPro / AMD DASH for PC clients?

They clearly mean “Windows machine,” not “personal computer.”

What's your plan for if an attacker gets access to your IPMI system and uses it to put rootkits on all your machines, including the ones that you image other machines from?

... or when that system gets hacked, like Mobile Guardian, where the hackers remote-wiped thousands of systems [1].

[1] https://www.heise.de/news/Mobile-Device-Management-Einbreche...

The IPMI network is airgapped

Pure speculation: The part where the Delta team didn't want Microsoft's help feels like it is probably that they knew that their systems were a mess of old, unpatched Windows clients and servers and were embarrassed about it (and that their legal teams probably knew that it was better to keep everyone out so they could sue later on)

Can anybody explain why an IBM server can be affected by a Microsoft Windows crash? Why did the IBM server need restoring from backup?

Also, wouldn't that contractually be IBM's issue to solve?

Well, they dint buy their Azure windows VDI for help desk and flight status displays.

> CrowdStrike followed up with Delta on the offer for onsite support and was told that the onsite resources were not needed

> Even though Microsoft's software had not caused the CrowdStrike incident, Microsoft immediately jumped in and offered to assist Delta at no charge following the July 19 outage," the letter said. "Each day that followed from July 19 through July 23, Microsoft employees repeated their offers to help Delta. Each time, Delta turned down Microsoft's offers to help, even though Microsoft would not have charged Delta for this assistance

Two vendors offer you free on-site help, during the biggest outage in history, and you turn it down?

This isn't the CIA, it's Delta Airlines. Bring those overpriced security monkeys in, make em sign an NDA, and have them sit there and watch you type.

To not do that... even if they were useless... I can't find a justification ....... other than being able to later claim it wasn't Delta's fault it was down so long, and nobody having evidence to the contrary.

Reply on news.ycombinator.com