Southwest Airlines grounds its entire fleet amid giant computer outage
nypost.com
nypost.com
On a serious note, it seems somewhat of an oversight that Southwest's entire multibillion dollar operation can be ground to a halt due to the weather pamphlet printing system breaking. Talk about single points of failure!
[0] https://www.southwestairlinesinvestorrelations.com/news-and-...
I believe they were also the most profitable US airline last year because of how streamlined their entire operation is. They could afford it if they wanted it.
Also, their entire fleet is made up of 737s, so it’s not like large airports get a massive planes (maybe a MAX vs a -700, but still not a huge difference)
Looks like their net margin was about 10%. Which isn't super low. But it's no 30% plus like many of the tech giants.
I had a flight with one layover. At the airport my first flight was delayed an hour, making the connection unlikely. I asked the customer service what I could do. Most alternative fights were booked, so I took a flight the next day with one very long layover. They said that because this issue was out of their control they were not obliged to provided a hotel room or anything else. I called the travel agency from my work and booked a flight from my original layover destination with a different airline. I then talked to customer service and changed my booking back to the original itinerary, thinking that the second flight may be delayed as well. Then the first flight was delayed by another hour, making the connection impossible. Thankfully I had a confirmation number with the alternate airline for the second leg of the trip. I flew to the layover destination. In flight I checked the Southwest schedule and saw the original connecting flight had been cancelled. When I landed I went to the other airline customer service desk to print my boarding pass. They said my ticket had been cancelled. I called the travel agency and they said that the airline was trying to avoid overbooking and was not letting any third parties buy tickets. There was another flight available with another airline, but the same thing: they wouldn't let the travel agency buy the ticket. I went back to the customer service desk and bought the ticket myself without any problems other than having overpaid for a flight for reasons beyond my control.
I'm currently sitting here, and will be for the next 7 hours waiting to go home.
There was a man with schizophrenia shouting earlier about needing to get home and not having his medication, and many minutes later I heard him screaming in the distance. Later there was another man shouting about masks. I also met a woman who was with her family trying to get to New Orleans, another couple going my way without a clue how they were going to get there, an old man and woman also going to New Orleans who were told their 30 minute layover (before the second delay) wasn't going to be a problem, and a young man who took it all pretty well and just took the next day rebooking and left to go find a hotel.
I often think about how my air-travel experience is different and stress free from others.
As someone that doesn't drive to the airport and can basically just walk through empty security lines, only books direct flights and doesn't have to wait till the holiday to travel, the stress is non-existent.
The worst common thing I encounter is my plane landing early at the destination and having to wait on the tarmac longer because the gate isn't clear yet.
In comparison, many others are driving to long term parking, taking long shuttles, have packed inefficiently resulting in long lines, rebalancing at both the baggage check area and at security, and have to compound the flight promptness gambles with short or long layovers. Can't tell which one is worse. Now that I think about it, many people likely don't have a way to get their ticket available easily either. The stress compounds on itself.
Did I mention the airport lounges? The discovery process can be nice.
Aside from my fundamental problem with security theater at the airports, the actual experience in the airport is not terribly bad but I have been stranded in places due to flight cancellations, I have experienced long delays, etc
Also I am a Big AND Tall person, for which the airplane itself I believe has been designed to purposely torture Big AND tall people, anything over a 2 hr flight and my knee,s and legs are in serious pain, the worst is then having to sit for 45mins at the other end "waiting for a gate"
Further given my location and my destination the majority of my flights are on small regional jets for which there is no "business class" to get a larger seat or more leg room, and "exit row" in these small jets is laughable
No, when ever possible I drive
Like...when you're in the security line, you should have your ID and ticket ready to show the TSA agent. I've seen way too many people get to that checkpoint and only then put their bag down and ruffle through it to find their wallet and ticket, completely oblivious to everyone that was in front of them in line and already had that stuff ready.
Then, when they have to put their stuff on the scanner...doesn't matter how many times the agents shout that laptops have to be removed from bags and their shoes have to be taken off, they'll try to go through with shoes on and leave their laptop in the bag.
Think about it, you are hurtling along a highway at 70 mph in a car with numerous components that are aging and can break catastrophically. All it takes is a tire blow-out and your car flips. Or the overweight truck driver in the lane behind you has a heart attack and his truck crushes your car like a soda can. Or a vehicle traveling 70 mph in the opposing lane loses control and flies across the median, hitting your car and killing you instantly before you even have time to react.
Meanwhile, the airplane covers that same distance in a fraction of the time and has not one, but two pilots, with numerous years of experience. If one of the pilots has a heart attack, the other one can take over. There are almost no vehicles on the road with that level of redundancy.
I'm in in the Portland, OR area. I would absolutely drive to Seattle (~180 miles) or San Francisco (~630 miles). LA (~1,000 miles) is the limit for how far I'm willing to drive.
But when I take a trip to theme parks on the east side of the country (Cedar Point, King's Island, and Holiday World were my last destinations), that's over 2,000 miles. That's 3 days of driving at a reasonable pace. I'm definitely flying that one.
Someone at Southwest must have looked at how much this was going to cost and panicked.
What I really don't like is that I didn't receive any notification about any of this. No email or text or phone call from Southwest or the travel agency. Had I checked HN I would have found out before returning the rental car.
I already check HN when a big web service is down. The time has come where we should check HN if our airline/train/public service is down too!
I loath it but this is one of the few circumstances where you should go full assertive. Don't take it out on the desk staff, they're having a far worse day than you, but stand there and make clear you are a problem that isn't going away until there's some resolution. Suddenly the hotel voucher will appear.
I want the whole system reformed. I've had to do this with United several times now and hate it so much I'll quite literally spend $500 more just to avoid them.
no Air Travel for me this year
I'm not a commercial pilot but would love if someone who is could post a plausible hypothesis how this would ground the whole fleet (short of a simple bureaucratic decision to shut down if everything isn't working exactly normally).
edit: s/FFA/FAA/
An important part of the flight plan is weather conditions along the route. There are obvious safety concerns, but weather conditions are also responsible for the route chosen, the amount of fuel onboard the plane, and passenger comfort. Without weather information, the flight plan cannot be filed and the plane cannot fly. Keep in mind that there are specific regulatory standards that must be met with any information included in the flight plan. A pilot cannot simply open up the weather app on their phone for instance. The weather info needs to come from an approved source.
I have no specific information about Southwest's current issue, but my guess is the third party service they use for weather in the flight plan is out of service. If they can't transmit this info to aircraft in the air or on the ground, then nobody gets to fly today. I'm also guessing that an incident like this will result in backup solutions being put into place, and that there is an IT or systems engineer somewhere who is trying to hide their "I told you so" expression.
Edit: an example flight plan can be found here: https://www.simbrief.com/system/guide.php#ofpsample. This is from a site for sim pilots, but the format of the brief is the same as what a commercial airline would file for a flight.
I also agree that, much like when the Space Shuttle blew up, there were engineers somewhere who immediately suspected exactly how and why.
In order to dispatch a flight you need to ensure that the weather at the arrival airport meets certain minima at that point in the future (hence the need for forecasts), and so does that at suitable alternative airports, and enough fuel is carried to be able to divert (depending on a heap of other factors such as aircraft load, capacity, prevailing winds enroute, at destination, time of day and so forth).
Aircraft are then also required to check the AWIS/ATIS or the Tower (usually the most reliable source of weather at an airfield) to ensure that the weather at the destination is within their safe margins.
There's the FAA requirements, and then airlines and aircraft and airports then also have their own established minimum requirements that all need to be within the safe range.
You can certainly dispatch a flight with potential updates and diversions as they progress / get closer to the destination, but with an outage and no means to update the enroute aircraft - there's usually no easy safe or legal way to dispatch them.
Also, the nice thing about wfh is the possibility of screaming about the stupidity of your organization more conveniently...
It also wasn't so long ago that scheduled domestic US airliners crashed on a fairly regular basis. That doesn't happen anymore because of advances in technology, this just being one little piece of that.
There are people complaining that they can't get through to customer service on their phone. Like, do they expect them to always have enough customer service capacity for when every single customer in the world has reason to call them? That'd be insane.
Flight cancelations are always a risk when flying. You need to always expect that you'll potentially have to be stuck somewhere for a night or two. If the airport floor doesn't suffice for you, fly airlines that have a track record of giving hotels for cancelations and have insurance/funds to cover a few nights if you aren't able to get a room from the airline.
This kind of stuff is unavoidable and just part of air travel. Be prepared, be patient.
I don't know with Southwest, but I know more than one airline has a system that sends automatic "choose one of these options" emails when flights are cancelled. When all of your customers need new flights, though, the issue isn't how the rescheduling is done, it's the fact that almost nobody is going to be able to choose an option they like.
Anecdotally, it has worked enough times for me to call/talk to an agent vs dealing with the automated suggestion system. Just don't be a jerk.
One automated system gave me a next flight of the following day. Spoke to an agent and was rerouted through another hub and only delayed by two hours overall.
In another case, a delayed flight caused me to miss my connection. Was officially told that I was out of luck and would have to wait (as were the jerks who asked) for the morning. I went and got a seat on the plane out that night.
I wonder if there just isn't a negative feedback loop here (everybody gets helped) so people just assume their way is the most effective.
Those aren't mutually exclusive. You can complain a lot but not be a jerk. You just have to understand and acknowledge the situation the customer service rep is in but still be firm on what you want.
Flight got cancelled? Send me a text, email, whatever. They have all of my information. Text me a link.
The reality is that they don't do that because they'd lose money this way. It's best to underprovide so people who don't want to wait just go elsewhere.
When covid hit last year I had to spend literally hours on the phone trying to cancel/rebook because Alitalia did not allow this online. The low-cost EasyJet instead *sent me an email* with a link to get a refund. It's easy. Nobody had to wait for disaster to hit to think of a solution.
These are just (anti-user) choices that companies make to save money. Some users will still call, but much fewer.
I don't know about Alitalia. Not all airlines have these systems.
Airlines always do it for me. Well, at least in the last handful of years I've been flying.
"I never understand people's expectations of other people's expectations..." Well, I guess I feel I understand that too.
"Be prepared, be patient..." Good advice, so is "don't have a global outage" but do people follow it? Nooooo
People on HN should understand better than anyone that this isn't really possible over a long enough period of time. It's not like Southwest has to ground its fleet every other month. This specifically is a very very rare event.
Should you be carrying a tent and three days food and water? That would add to your carrying ons. The only thing I can think of is not making appointments on the other side where your life is screwed if your plane is diverted. But how much you can do that or need to do that varies by person.
It's the Louis C.K. bit about being (roughly) 'in a chair in the sky at 500mph: it's a miracle yet no one's happy'.
Louis is right about You're flyiinngg! but also flying is normal now, so we always hope that things get better. Rich people don't even fly anymore, they sleep in a luxury hotel in the sky. Things are possible, it's just that they're not cheap yet.
Gogo, the one that absolutely sucks, locked many airlines into a 10 year contract so they're screwed until that ends. That's the only thing stopping the fast service(s) from rolling out.
Then boom! Back to NoGo, limited craptastic, swore I would never pay for it, service.
Yes, why wouldn't they, they just canceled their flights.
In the US those are? United, Delta, and American for the most part don't. They'll look for any reason to claim WX issue so they won't pay out.
FWIW I had United give me a hotel in 2019 after a mechanical issue.
These are people stuck in strange places who are anxious and uncomfortable. Some of them are probably going to miss important things. Others have to worry about kids/pets/etc. That's not to say they shouldn't be patient (and certainly they shouldn't be taking their frustrations out on the airline staff), but you're demanding rationality from people in the exact sort of situation where it can be really, really difficult for someone to be entirely rational.
They're trying to do what they can (call customer service) and find themselves helpless - that's not a situation where they're going to reason through the economics of call center staffing.
I'm saying it from the perspective of someone who has been stranded at an airport. It's very easy to chill tf out for a while and wait in line for a while at the service desk to see what you've got to work with. You're not dying, it's not literal fight or flight. You're just stuck for a while.
> Your points are valid, but I think the irony is your expectations aren't reasonable.
I don't think it's reasonable to scream at service reps, which I guarantee will happen to nearly every single Southwest rep today. I've seen it so many times.
And it's the people who are confused why the customer service lines are jammed, those who are irrational, who are more likely to be the screamers. That's mainly who my comment was directed at.
Those people need to relax a little bit.
Just a generation ago they could have been driving down a remote road, in an unreliable car with no cell phones.
What would they do then?
Well they'd start walking and nobody on earth would know where they were.
Being stuck at an airport, you have something in your pocket you can use to inform everyone of what is going on, handle getting your pets taken care of, etc.
That said. I think people were understandably pretty pissed. I talked to a family that was taking their first vacation in 18 months. He only gets a handful of vacation days a year and due to cancellations their vacation has been ended. Just like that.
If I was them I'd be out of my skull. It'll probably be another full year before they get to try again.
If you traveling for work.. no big deal Company may be out some money, you may be delayed for a meeting, etc etc
If you are traveling for personal... these things are not as easy to just rescheduled, push back, or change. The family likely also lost money on reservation especially if it was a "vacation package", which likely use the same excuse as Southwest so screw the family out of their money.
Different airlines handle misconnects differently: Delta sometimes auto rebooks, United and US Airways required you to call and rebook.. although US Airways required you to work with the gate or call before hand and you got issued a new ticket at the kisok. On top of this.. there are different rules you aren't explained about what conditions indicate you can break the ticket without being penalized. (Schedule change being one)
Lastly, theres the effect of that the airline never takes responsibility when they screw up. It's always "eh you should know better" .. Leftouthansa sends your bags to EWR on a FRA-LHR-ORD flight? Maybe you'll get your bags in 48 hours? Don't forget to ask for an amenity kit (that's on you to ask for). Bag got lost and you want reimburstment? Please keep your recepts from years ago for the jeans you had in there.
Sure weather is hard to predict but it is good enough to alert you the 3-4 hours before your flight so you would know prior to arriving at the airport. Computer systems can be made fully redundant, etc.
I agree that it's pointless to expect customer service to handle the load during these crisis precisely because these events should either be one per decade type events or at least be handled in a planned, graceful manner.
What would that look like? For every flight let customers at least check what the probability of delay/cancelation is - the airlines certainly have a number.
When a flight is canceled; book first and business class on alternative airlines/flights. Let economy/other customer's select alternative flight or request refund via zero human/robocall interaction process. If more than 50% of the flight's capacity has already checked in then alternative lodging should be mandatory and meals should be provided.
It's painfully obvious that the airlines know that as long as all of them provide terrible service then none of then will be expected too.
Years ago, they compensated for this with generally lower prices. However, in more recent years, I've noticed their pricing isn't nearly as advantageous compared to competitors. Yes, if you're one of the first 20% to book a flight six weeks in advance, and can accept little flexibility in terms, and are willing to go at a less desirable time, there are still deals to be had. But in any other scenario I've found United, American, Delta, etc to sometimes actually be cheaper, which was previously rarely the case.
Who's on first. What's on second. I don't knows on third. Where are the clouds?
In a way people probably think things like that are insane but if you step back and look at it, if it was getting the job done all these years it was decent engineering.
I was paying attention after that.
An airline that can’t fly planes due to a third-parties weather app. (Which, for all we know, is dependent on other third-party apps.)
These problems will grow exponentially.
Just wait until Company A goes offline because it relies on Company B, which goes offline due to Company C, which (and this is where the fun begins...) goes offline because of Company A.
I guess what you’re saying is what if the weather app crashes both Southwest and Kayak. And once the weather app goes back online if Southwest and Kayak were both dependent on each other then they couldn’t recover even though the original cause had been resolved.
Of course, in the ‘real’ world, these kind of interdependencies exist all the time. The difference is the speed and scale with which these glitches perpetuate.
I mean, you had the same issues in the 1800s with a supply chain. But, a lumber mill in Canada that burns down would not have instantly caused houses to collapse in S. Carolina.
Nowadays, the problems happen fast. And, with brilliant IT types, the problems get resolved fast.
But, boy howdy, I get the feeling — just from reading the angst-ridden accounts from overworked stressed out techs types on HN — that we may be approaching a point where the ‘fixers’ are going to get overwhelmed by the bugs.
Listen, I am sure this is a rigorous, safety-critical procedure. Several comments already show very well some details about flight plans and contingencies, and so on.
So how in the flying fsck are there not multiple service providers for a piece of mission critical data like this? Shouldn't this be something provided by multiple vendors over multiple ISPs and hosted in multiple regions by multiple cloud providers?
How much money did they lose in the past two days because they didn't think this could happen?
Might?
We rapidly need to reframe our security posture.
It would suck to have IT systems intentionally disrupted but I don't think even modern airliners can plausibly fall out of the sky even if they suddenly lose contact with all ground based systems.
I think there are procedures in place even if air-traffic control systems, comms or GPS fail. Last time I checked, pilots can still fly the plane with their eyeballs, on-board radar, compass and ground speed. In fact, I recently saw an article where some senior safety pilot was concerned that with all the nifty automation, today's pilots aren't getting enough practice doing, you know, actual flying of the airplane.
isn't that one of the more benign scenarios? staying connected but getting fed malicious incorrect info seems much more dangerous.
Like that twitter acc that collects stuff like a shoe that needs firmware updates
Commercial flight safety is all about standardization, reproducibility, and predictability. Switching the weather source for the pilots violates all three of those principles.
Somehow we've decided that any loss of life or injury via a crash is unacceptable. Yet we treat delays (both airline and traffic) that cost millions of human life hours as barely newsworthy.
Would it be better if we considered policies in terms of human life hours saved rather than human lives? Is that already being done?
Taken to an extreme, suppose we found 5 hours worth of airline safety video prior to a flight saved 10 human lives per year as it made it less likely people would evacuate the plane with their belongings.
If you led the FAA, would you mandate each passenger watch this video before each flight in order to save those lives? Put another way, is 5 hours * 5 billion airline passengers / year worth 10 humans lives / year?
People who die in a plane crash don't get to execute on plans A through Z.
Even if you run the numbers as a comparison of hours lost waiting for your plane to years lost from somebody's actuarial life expectancy because they died in a crash, it's not a one for one comparison.
Meanwhile, passengers largely assume they won't die regardless of which airline they pick, and happily take absurd delays and unnecessary connections in exchange for lower ticket prices time and time again.
In other words, wouldn’t the calculation need to be treated much like any capital investment with net benefits needing some time to kick in?
For whatever reason, we as a society have decided that airliner crashes are scarier than car crashes. Just like we've decided terrorism, nuclear energy, and self driving cars are scarier than they actually are. Should we be this way? Probably not, but this is the way the world is.
My question though is how should it work? What's the most optimal tradeoff between delays vs lives lost?
Many other factors contributed to the sinking, but one of the NTSB bulletpoints was to rely on official weather sources only. I wonder if that directive played a role here.
https://news.ycombinator.com/item?id=15554602
Everywhere else it is 150k-200k a year with the bare number headcount and you are a cost center and treated like one.
So not surprising to me that every company in the world that has a large physical plant, so already has large single points of failure, doesn’t have redundant data centers or cloud based systems.
Given the number of hacks anything online literally can’t be secured, so they are damned if they do and damned if they don’t.
One answer is to go back to analog systems, which ain’t happening!
Because that's what it is - if your entire fleet is going down and you could have spent more to hire engineers, you just are irrational at pricing the value of engineering labor.
It’s always easy in hindsight to say they should have spent more, but opportunity cost is very real, and margins are very slim.
That's an odd line of thinking. Other airlines have catastrophic outages too. Just at different times/dates. They do sometimes share 3rd party dependencies, but that happened to not be the case this time.
Mediocre software is eating the world
A cost is a thing you need to pay but if you spend more on it you get very little benefit. Thus it has a binary nature or at least a nature where "good enough" is all you need. A profit center is something where if you make it better you generate more revenue. Therefore things like necessary infrastructure is a cost, whereas adding more routes is a profit.
Whereas I think you are confusing cost and profit to mean "important" and "unimportant". Not having your headquarters fall down is important. Having HVAC for your office workers is important. But no one is going to say "Let's choose this airline, have you seen how awesome the HVAC in their main office is? And how the foundation to their HQ is going to last a hundred years longer than their competitor?" But they will say "Let's choose this airline because it has a direct flight to where we want to go".
Seems to me the MBAs ran the numbers a while ago, and here we are.
Frontier was dirt cheap, so ok whatever, but Southwest basically chargerg 535 bucks for a 2.5 hour flight from major city to major city and then didn't even let people drink on it. They even gave us free alcoholic beverage coupons that were then invalidated. Complete bullshit.
Definitely one of my top 5 least favorite flying experiences. Flying is such a shit show these days. It used to be so calm. I can't believe airlines can charge half a g and also not let you drink. I get stressed a bit during flying and they just chuckled and kept walking. Fuck the airlines. It is egregious that this covid culture has resulted in this insane fear of interacting with other people.
I think when I move to Miami in September I am not going to fly, I'll take a train with the highest possible class just to get around the absolute garbage experience that flying has become. It'll take 24 hours but maybe, just maybe, it won't be Kafkaesque
New statistics from the Federal Aviation Administration show a rapid rise in plane rage incidences.
The FAA says it’s gotten 1,300 reports in the past three months, as the number of passengers remains below pre-pandemic levels.
...
In a typical year, the FAA sees anywhere from 100 to 150 cases - only a fraction of those seen since February.
Source: https://www.msn.com/en-us/news/us/faa-reports-rise-in-plane-...
1) So what is wrong with all these psychos?
and
2) and what does any of this have to do with drinking? All of these cases seem to be people either retaliating against former employers or just not wanting to wear a mask. Banning drinking seems to be a band-aid that doesn't solve the issue at all, and doesn't even address the core issue at hand.