I read somewhere that their crew tracking software was hit hard and took time to recover. Will look for source on that.
(Edited) source: https://news.delta.com/update-delta-customers-ceo-ed-bastian
“… and in particular one of our crew tracking-related tools was affected and unable to effectively process the unprecedented number of changes triggered by the system shutdown…”
As someone else pointed out, they probably weren't ready by the time they needed their systems for the morning rush so they went to their business continuity strategy (manual). This has a throughput and recovery time penalty and obviously it compounds the longer they are in that mode.
I think what we're finding with the Southwest meltdown and now the Delta meltdown is that the big airlines just don't have the manpower or scheduling slack to accommodate going into business continuity. I do think this should be investigated. Hopefully financial penalties incentivize action but time will tell.
> big airlines just don't have the manpower or scheduling slack to accommodate going into business continuity
Which one profits the CEO more? Stock buy-backs or robust IT? Robust IT is only good for the company in the long term; however, with stock buy-backs or other skimping on IT, if disaster like this happens, the CEO just takes his golden parachute and leaves, but if no disaster happens, he gets a huge bonus to buy another private yacht.
Delta already took a huge financial hit for this.
It's not so much a severity as "hard"; but with the hub and spoke model that Delta uses, scheduling being down (at all on Friday), combined with FAA hour limits. It becomes exponentially difficult to reschedule flights.
Put more plainly, on Friday, your scheduling software is down for 4 hours in the morning, so you "borrow" any replacements you need for employees that are late or sick. This ruins the availability for the next flights, at which time you hope the system is up again; but if it's not, you borrow from the evening flights. Combine this with each flight that was late/cancelled as you were hoping to fill now affects the hours available for the employees that were available. Finally, as you've cascaded this, you head into a weekend trying to catalog how many hours each crew member did or did not log, and you're not sure how to get them back in time.
Otoh, Delta seemed to have recovered after about a week, and canceled about 1,000 out of about 4,000 flights for several days. It's way better to fly 75% of the daily flights than not. There's less wiggle room in a summer schedule for weather, but there's still some wiggle room.
I read somewhere that their crew tracking software was hit hard and took time to recover. Will look for source on that.
I heard on the radio (maybe NPR, not sure) it wasn't about the computers, it was about Delta's response.
According to the report, the other airlines delayed flights, while Delta cancelled them outright. That left Delta with more people and planes in the wrong places, making it harder to recover.
[0] https://en.wikipedia.org/wiki/2022_Southwest_Airlines_schedu...
Keep in mind there was no way to opt out or delay CS Channel updates.
It would be like claiming raid1 is a backup.
Any DR system will have to accept some risks, and those don’t necessarily invalidate it in general, just make it insufficient for some scenarios.
Conversely, if they ran the main system on windows with crowdstrike and the DR one on poorly configured linux with no security software, they probably would have needed more sysadmins, had more trouble maintaining software for both, and been vulnerable to risk from both linux and windows bugs, so I feel like they made the right tradeoff in general.
I’m sure you, who can deride this DR system, have devised your own system such that it is resilient to a meteor destroying the earth.
That reminds me one of Corey Quinn's comfortable AWS truths.
https://x.com/QuinnyPig/status/1173371749808783360
> If your DR plan assumes us-east-1 dies unrecoverably, what you're really planning for is 100 square miles of Northern Virginia no longer existing. Good luck with that ad farm in a nuclear wasteland, buddy!
Do CS updates somehow work over airgaps? You know, the kind that production systems have to prevent any access to or from external networks? Well... some production systems anyway.
It looks like it wasn't a good security product after all...
https://www.tomshardware.com/software/windows/windows-31-sav...
https://www.forbes.com/sites/tedreed/2024/07/20/meltdown-wha...
> A story on the website govtech.com on Friday asked the question, “Why isn’t Southwest affected by the CrowdStrike/Microsoft outage?
> “That’s because major portions of the airline’s computer systems are still using Windows 3.1, a 32-year-old version of Microsoft’s computer operating software,” the website said. “It’s so old that the CrowdStrike issue doesn’t affect it so Southwest is still operating as normal. It’s typically not a good idea to wait so long to update, but in this one instance Southwest has done itself a favor.”
The govetech.com article is https://www.govtech.com/question-of-the-day/why-isnt-southwe...
which linked to https://www.digitaltrends.com/computing/southwest-cloudstrik...
which linked to an earlier Forbes article - https://www.forbes.com/sites/hershshefrin/2022/12/31/can-sou...
> The December 2022 scheduling fiasco was the result of skimping on information technology. I am old enough to remember when Microsoft introduced a new operating system called Windows 95, to replace its predecessor operating system Windows 3.1. The 95 in Windows 95 refers to the year of its introduction: 1995. By some accounts, major portions of Southwest’s scheduling system for pilots and flight attendants is built on the Windows 95 platform. That platform is now more than 25 years old.
“That’s it. That’s where all these stories can trace their origin to. These few paragraphs do not say that Southwest is still using ancient Windows versions; it just states that the systems they developed internally, SkySolver and Crew Web Access, look ‘historic like they were designed on Windows 95’.”
https://www.osnews.com/story/140301/no-southwest-airlines-is...
I am not claiming that they run Windows 3.1 or Windows 95 ... but rather "this is where that story was sourced from" because everyone kept linking to somewhere else. The relevant XKCD is https://xkcd.com/978/
Essentially modern versions of Soviet-style disinformation campaigns, but augmented with new technology (social media), and without the ideological hindrances of a Communist government (e.g. sell hard to both Right and Left).
RAND Corp calls it "the Russian Firehose" model: https://www.rand.org/pubs/perspectives/PE198.html
Similar approaches are also used by NK, Indian, Chinese, and other national-tier disinfo campaigns. This contrasts with models used by the West, which are often less about creating a disinformation clusterfuck, and more of a "watch our Disney / BBC / Scandinavian TV & movies and their implied messages about freedom and human rights and shit".
Obiously Russian, Chinese, and all other governments engage in information campaigns, and obviously the US government knows a lot about what they are. But we the public does not necessarily have the same information. It's not really possible to distinguish the "well known and documented (by the military and espionage industrial complex)" operation of foreign countries from domestic propaganda developed by those corporations and agencies to influence their own citizens.