Southwest Airlines pauses departures after reporting technical problems | Hacker News Reader

Southwest Airlines pauses departures after reporting technical problems

cnbc.com

75 pointsjbredeche26 comments

I love how they immediately place blame on the vendor. You would think an operation of this scale and importance would have some redundancy built in or after the previous disaster would have attacked single points of failure.

Where do they "place blame on the vendor?" What SPoF should they have addressed? They blame "a vendor-supplied firewall," which is different from the scheduling software that caused the incident over the holidays, and different from blaming the vendor itself.

Are we reading the same article?

“Early this morning, a vendor-supplied firewall went down and connection to some operational data was unexpectedly lost,”

They're blaming the vendor and absolving themselves of any responsibility. If losing a connection causes catastrophic failure, who's really at fault?

True, but being technically accurate isn't bad. If a vendor owns it, a vendor owns it.

Not having HA on a network segment is bad, though.

Can we validate the accuracy? Probably not, but I can say in my own postmortem or RCA analysis taking zero responsibility or deflecting is a sign of a greater problem. There is always room for learning and growing.

You can't claim they aren't taking responsibility unless you've seen the internal postmortem. External messaging is PR. It's designed to not affect the share price. You can't extrapolate anything about internal process from it.

You seem to be pretty invested here, everything okay? It's shit PR, devoid of any responsibility, and they're already in the dog house when it comes to stock value. For me personally, I just won't fly this airline, I can't trust them on the ground why would I put my life or valuable time in their hands or reward them with business?

We don't need to a postmortem, the blame has been solely laid on someone else's shoulders. I think the only thing we can extrapolate from this conversation is that you possibly work for SW, or that you're upside down on SW shares. Have a great day and welcome to HN.

Saying a fact isn't deflection. I have been in post-mortems too, and accuracy is important.

If I work at a call center and there is an outage on a partner's PBX, then the summary will say "vendor PBX caused an outage". Part of that conversation will be "how to we mitigate the risks of a vendor PBX outage in the future", but the technical fact stands: Someone else's equipment failed.

That's a description of what happened. It doesn't say why it happened. Nobody is being blamed for anything. For all we know, somebody at Southwest tripped on a power cord and shut down the machine running the firewall. Facetious, yes, but I have once had someone literally trip on a power cord for a machine running test software I cared about, shutting it down. (Yes, we fixed the issue and got UPS's for these machines after that lol)

At a minimum, that firewall appears to have been a single point of failure.

That's how security stuff intrinsically works. It sits in the critical path and fails closed. If it failed open, then it would be "malware installed on Southwest's production network due to unpatched node.js module" or something.

>> [security stuff] sits in the critical path and fails closed.

Not entirely accurate. Some inline IPS appliances fail open, and networks can be architected to fail around a single firewall in a high availability pair.

That's a dramatic over-simplification. You have redundant / clustered firewalls that allow for testing configuration changes before they're deployed everywhere so you never end up in a situation where you have to "fail closed". You have redundant links into critical data centers, either hot/hot or hot/warm. You have redundant power supplies in the racks running the firewalls and routers. You have redundant data centers (where necessary).

Nothing stopping you from running redundant links with redundant appliances, except beancounters ofc

I guess I've accepted that it's something intrinsic to the software that's going to make it fail. Bad config push, bad packet segfaults the packet inspection software, etc. (Yup, people still write parsers for untrusted user input in C, and don't fuzz it. That's the state of the industry.) In those cases, all of your replicas fail identically at the same time.

For mere hardware failure, we seem to have moved past that as an industry. Cloud providers will "transparently" migrate your app to a new computer if the one you're running on blows up. (Transparently is in quotes because you'll see the P99.9 latency increase when it happens.)

Network security devices are often deployed with high availability baked in. If one firewall appliance dies on a production system, it should have a peer to failover to.

We don't know what happened, but at least for hardware, HA is the standard.

Interesting approach. What does it even mean?

Wouldn’t most of an airline run on “vendor supplied $thing”?

“Today, a vendor-supplied plane went down […]”

If you have a critical bottleneck and you don’t have a spare you made a mistake. A couple.

It reminds me of the saying: "when you owe the bank $100 that's your problem; when you owe the bank $100 million that's the bank's problem"

It doesn't matter if fault lies with an external vendor, people can't fly and Southwest is still going to take reputational damage as a result.

The way I’ve heard it, which I think is a bit punchier is “If you owe the bank $10,000 dollars, the bank owns you. If you owe the bank $100 million dollars, you own the bank.”

this is why I walk everywhere - no tech debt!

An earthquake is a form of tech debt.

I think.

This is their second time if I'm not mistaken. It's sad that there's no changes to prevent this.

> Southwest had a meltdown over the year-end holidays, which drove thousands of flight cancellations and resulted in an $800 million pretax hit for the fourth quarter. The airline upgraded scheduling software that couldn’t handle numerous changes to crew assignments during severe winter weather late last year.

> The issues Tuesday did not appear to be related to that software.

From the article, emphasis added.

Unrelated except that the same ethos and processes probably led to both.

https://news.ycombinator.com/item?id=34149340

Reply on news.ycombinator.com