Southwest’s meltdown from a pilot’s perspective
facebook.com
facebook.com
> But as time went on the operation began to deteriorate. There was little investment in upgrading technology (after all, how do you measure the return on investing in infrastructure?) or the tools we needed to operate efficiently and consistently. As the frontline employees began to see the deterioration in our operation we began to warn our leadership. We educated them, we informed them and we made suggestions to them. But to no avail. The focus was on finances not operations. As we saw more and more deterioration in our operation our asks turned to pleas. Our pleas turned to dire warnings. But they went unheeded. After all, the stock price was up so what could be wrong?
I see this in tech all the time. It is indeed very hard to measure the return on investment for tooling and infrastructure in tech! Any infra work, whether it's splitting up your monolith into components, improved developer tooling, or fixing flaky builds has a vague and hand-wavy return on investment and has to compete with "Well customer FooBar will sign a $10M contract in Q2 if we build feature BashBaz instead," and now good luck as an engineer explaining how and when exactly your investment in developer tooling is going to make the company $10M in Q2.
But if you never invest in these, then the machine comes to a grinding halt and everyone's hair is on fire, and there is no quick fix because the solution was to invest in the problem proactively starting 2 years earlier, and now that you're so late it's going to take 4 years to solve the problem because you have to first deal with all the debt you accrued.
This is a place where leadership really can have 100x impact by establishing a dual-route for evaluating projects. The first route is the traditional "how much money will this make us", but the second is based on principle ignoring $$$ returns, "what is the most impactful operational improvement that our front-of-the-line staff is recommending?" You have to simultaneously run both strategies!
Agreed. I really see no reason you can't do this at any scale. We do it today in our tiny startup. The strategy we employ is a little different - Some of us "secretly" work on product iterations so as to not distract the rest of the team until we are sure we've found something. If the new work doesn't pan out, no one even hears about it. If it's amazing, we sort out a gradual introduction of the idea to the team.
There is no reason you have to have 100% of the organization bought into a rewrite/refactor/iteration of the product. All it takes is for one or 2 strong employees to go off into the dark forest for a while and come back with a potential prize. For an organization as large as Southwest, this would be a negligible cost even factoring in their razor-thin operating margins.
I wish there was a more prevalent culture! My most productive work comes about this way, and I've yet to find a job where I'm able to do this regularly.
My experience is that most engineers don't try, because they don't tend to speak finance. A lot of developer tooling is roughly quantifiable; this tool saves X minutes while doing Y, do a little survey to see how often people do Y, multiply Y and X to get time saved per year. Divide X*Y by the number of work minutes in a year, and you have the number of man-years saved. Multiply that by the average salary of an engineer, and that's very roughly how much the dev tooling is "worth". It's not a precise answer, because 50 employees saving a couple hours of time a week isn't the same as having another employee, but it's a workable approximation.
> the solution was to invest in the problem proactively starting 2 years earlier, and now that you're so late it's going to take 4 years to solve the problem because you have to first deal with all the debt you accrued.
This can also be roughly quantified. Get rough numbers on the resources required to fix the problem now. Look at what's coming for the next year or two in terms of projects and infrastructure that will exacerbate the issue, figure up roughly how much effort it will take to fix those. Convert that effort number into salaries, and you have a rough idea of what the cost of delaying the fix is.
If you're worried about tech debt causing a catastrophic failure, figure out what that scenario looks like, calculate up the concrete costs of the failure (SLA violations, refunds, etc), then have another column for potential costs like regulatory fines, and a third column for things that have costs that the finance people are better suited to handle (what's the cost of reputation loss?). Again, it's unlikely to be completely accurate, but it gives you a data-based platform to speak from.
Even better is to engage the finance department to help draw up those numbers. Money is their forte, and calling them in to help with money questions is the same as calling in the frontend people to help with React or whatever.
I think sometimes we engineers spend too much time around other engineers and get too comfortable speaking in terms of engineering priorities. Those don't extrapolate; finance and legal don't give 2 shits about "flaky unit tests". We have to convert those engineering priorities into other teams' priorities if we want them to care. Finance cares about money, so rephrase the engineering problem into a money one. Legal cares about liability, so rephrase the problem into a liability.
Rather, the people who should be running things are those that know how to do the core work of the company. For most companies this is not finance. For an airline, this means that the software department should be speaking the language of operations and logistics when justifying how the new system will be better and what it will look like when the current one fails.
The rub here is that finance,lingua franca /secret handshake of the executive caste, demands proportional compensation on top of large salaries for their "contributions".
If I as an engineer make the company a billion or save it two million, do I get the same compensation awards? Yeah no I'm in the peon class.
I'd get a plaque or maybe "distinguished" in front of my title.
You've got to speak to decision makers in their language. If your CEO is from finance, speak finance; if they're from engineering, speak engineering (and you bet the finance team will learn to speak it too); if your CEO is from sales, speak sales; etc.
A great CEO is going to learn to speak all the business languages, but even still it helps to speak their native tongue.
I love the commentary about investment in operations. They hire a finance-
oriented CEO who doesn't invest in building out tooling and infrastructure.
Instead of pulling bag carts up to planes with a tug....you had to PHYSICALLY
pull them up to the plane.
They did not scan luggage and MOST WN workers were against getting scanners
because they felt it would take jobs. Hand counting bags, freight and mail
in 2017 !!!!
They don't have load planners. So they use a bungee cord at the cat and an
ops agent lowers it down to the Ramp agent.
do we have proof this CEO declined any attempt at internal investment in tooling/infrastructure? is he solely to blame?
why did a board of directors/shareholders/C-level executives stand by and let this happen? how can everybody be making $400k/yr-$20m/yr and be level 101 “summarize it in a one sentence” incompetent?
The board sees operational expenses as a line item on a slide, they don't see the complaints from front-line workers or middle management. Infrastructure is one of those things that either works fine or fails catastrophically, and you need a series of those catastrophic failures to show a dent in the quarterly/YoY results, so it's unfortunately also easy to ignore if your metrics are finance-oriented.
You can also get into a culture spiral, similar to what is happening at Boeing, where the management is gradually replaced with folks who focus on financial results and promotable metrics. This is ultimately the CEO's responsibility, but they need to install the right folks at the higher levels to fight against this tendency.
I just don't like to believe that something as sophisticated as a $20b company can be boiled down by over-simplified observations you and I make from the comfort of our desks.
There's got to be something more complex.
https://www1.salary.com/Robert-E-Jordan-Salary-Bonus-Stock-O...
CEO is making $3m/yr and you and I are supposed to be able to detect his weaknesses/mistakes/pitfalls at the blink of an eye? If it was that easy, why don't they just "not make that mistake" and fix the problem?
I just don't like to believe that something as sophisticated as a $20b
company can be boiled down by over-simplified observations you and I
make from the comfort of our desks.
The excessively hands on recovery plan would've been manageable at a much smaller airline.
From Southwest's little PR piece on Bob:
During his more than three decades at Southwest, the
airline grew from a regional presence with around 7,000
People and 90 airplanes to a beloved national brand
with more 54,000 People and a Customer base that topped
130 million Passengers in 2019. Bob has been responsible
for the delivery of many key Company initiatives,
including executive oversight for development of the
Southwest.com e-commerce platform, the revamp of the
Rapid Rewards® loyalty program in 2011, the $3.2B
acquisition and integration of AirTran Airways, the
comprehensive "Heart" brand refresh in 2014, significant
domestic and international network expansion, and the 2017
switch to a new reservations system.
The reason CEOs are paid so much is because they are tasked with the hard part of that problem; e.g. they have to find an operational strategy to create profit in a complex parameter space.
Wonder how many software engineers at Southwest immediately sold their vested shares and bought real tech stocks (like FAANG). They must be laughing right now!
I agree that it's hard to quantify those kinds of things, but I might quibble over the degree of difficulty. At the very least, some techniques for dealing with these kinds of "hard to quantify" scenarios are known, published, and have been analyzed to varying degrees. For a company the size of SouthWest, with the amount of money that's at stake, I find it surprising that they wouldn't use such methods (or maybe they do and just still managed to screw the pooch, I don't know).
What means am I talking about? Well, you could start with by the book Scenario Planning[1], and then have domain experts assign costs to different scenarios and use calibrated probability estimation[2] to assign probabilities to those scenarios. And then you could use Monte Carlo simulation[3] to get an idea of the likely distribution of outcomes. From there you can calculate a range of likely costs (or returns, depending on which way you frame the problem) for dealing with, or not dealing with, specific scenarios.
This is a very high level and hand-wavy description, btw, of the methodology Douglas Hubbard[4] lays out in his book How to Measure Anything[5].
[1]: https://en.wikipedia.org/wiki/Scenario_planning
[2]: https://en.wikipedia.org/wiki/Calibrated_probability_assessm...
[3]: https://en.wikipedia.org/wiki/Monte_Carlo_method
[4]: https://en.wikipedia.org/wiki/Douglas_W._Hubbard
[5]: https://www.amazon.com/How-Measure-Anything-Intangibles-Busi...
It’s easier to wait for them to fail so that it’s abundantly clear to everyone that they need upgraded and improved, and everyone is instantly on board with doing so. Yeah, you take some heat in the immediate aftermath but in the long run it’s probably cheaper and less risky all around.
Alternatively, those systems may never fail, or take a long time to do so. it makes counterintuitive business sense.
> Southwest continues to encourage and applaud out-of-the-box thinking from everyone at the company, from flight attendants to top-level executives. He recalled one mechanic, for example, who submitted a sketch showing how to fit an extra seat onto the aircraft, in order to compensate for some over-wing exit seats lost due to a new federal air safety regulation.
> In general, Kelleher said, if a Southwest employee submits an idea like that, he or she can expect to get an answer back within a week, and a lengthy explanation to boot. The point, he said, is “never spoil an idea, because when you spoil an idea, you ensure that you’ll never get another one from that person.”
[1]: https://www.gsb.stanford.edu/insights/herb-kelleher-manage-g...
Of course this is in no way specific to the airline business.
Are funds the problem? Has investing just become too abstract yet essential for me?