Outsourcing Cost Boeing Billions (2019)

A key factor in such failure is misaligned incentives. A team in a company is more likely interested in solving problems for the longevity of the company, so they think long term and they iterate. They also learn about the domain over the years. On the other hand, outsourcing companies just want to get a project done in time and under budget, and their hire the least qualified people to just meet the negotiated bar. To that end, there's really no incentive for an outsourcing company to hone any kinds of craftsmanship (there are exceptions in small and specialized consulting firms, of course). Plus, there's always information loss and distortion when a company communicates their needs to an outsourcing vendor. Consequently, all hell will break lose when a project is complicated enough.

It is not necessarily outsourcing fault but rather “people is a resource” mentality. This mentality leads to outsourcing to cheap locations because they are cheap. But I’ve seen in other context as well.

So they'd create a subsidiary at the location of the contractors, which is already a standard practice...

And I already have a plan for doing so: https://gavinhoward.com/2024/06/a-plan-for-professionalism/ .

[flagged]

The 737-max was aeronautical malpractice that Boeing tried to band-aid with software.

https://spectrum.ieee.org/how-the-boeing-737-max-disaster-lo...

And the fact that they wanted to spend nothing on the software and then rush to production (this was exposed in this article) is some cluster-b antisocial personality disorder shit...The bean counters had the stock churning at all time highs and they were going to do things their own way (trading at 450 per share shortly before these tragedies).

So it's HIGHLY unlikely that Boeings software for the 737 Max would have been operating at spec given the use-case (overcoming hardware problems with software in a way that's never been done before). MCAS was not designed to take into account that external sesors could be out of wack (which happens all the time). MCAS took liberties and had opinions that ran counter to the norms of aviation. And this is the hallmark of poor software engineers that have no domain expertise and no ability to push back against anti-social personalities masquerading as managers.

" When the two computers disagree, the solution for the humans in the cockpit is to look across the control panel to see what the other instruments are saying and then sort it out. In the Boeing system, the flight management computer does not “look across” at the other instruments. It believes only the instruments on its side. It doesn’t go old-school. It’s modern. It’s software.

This means that if a particular angle-of-attack sensor goes haywire—which happens all the time in a machine that alternates from one extreme environment to another, vibrating and shaking all the way—the flight management computer just believes it. "

If you want to have the title of SR. Engineer and lives are in the palm of your hand you better be prepared to be the bad guy and take on management when they're being driven by motivations that run counter towards the quality and the usecases for your code. VW had a similar culture and those "senior software engineers" went to prison, not the managers (remember that).

This. On top of the MCAS modification being necessary in the first place.

Higher torque generated because of more off-center thrusters surely raised eyebrows in the stall risk mitigation team.

In the end, that's 3 teams where management made the topic being fixed by another team. That's systemic. And with a single confounder: cost measures being prioritized over safety.

With the current QA problems, it cannot be made more clear that that single culture element is the root cause at Boeing. And ethics dictate that no more experiments be made, and the top 200 of management be forcefully removed.

Apparently 1/3 of all software vulnerabilities represent design weaknesses which were introduced in the requirements phase. The MCAS flaw seems to belong to this category which you describe.

source: https://insights.sei.cmu.edu/blog/a-tool-to-address-cybersec...

Separating controls from SW is a strange thing to do. The issue here is a culture of shortcuts and awful money-driven decisions, not any particular discipline, though delayed software fixes did cause more accidents, and it was poorly designed overall.

Ultimately, the MCAS relied on a single sensor, one which was known to fail, and only displayed the redundant sensor data - get this - if they bought the additional option to show when the sensor failed.

> However, whereas MCAS was activated automatically, without pilot action, the cockpit crew would have to notice and act on an AOA DISAGREE alert. Further, the AOA indicator and disagree alert were not standard equipment on the 737 MAX, although the AOA indicator had been on earlier models. Boeing offered them as “add ons” at additional cost

Without that "add on" you'd never know the MCAS was acting on faulty data. And all the while management worked overtime to mislead regulators on the potential impact of MCAS to avoid additional scrutiny and training requirements. So nobody knew the safety of their aircraft depended on that non-standard equipment package.

Yeah - total business and design mismanagement driven by greed.

---

https://mitsloan.mit.edu/teaching-resources-library/boeings-...

> When I say I changed the culture of Boeing, that was the intent, so it’s run like a business rather than a great engineering firm. It is a great engineering firm, but people invest in a company because they want to make money. --Harry Stonecipher, 2004

I think that outsourcing was a symptom of this disease. Not MBA cancer, not poor software, not anything other than a deliberate cultural shift which led to all those other things.

Check out this: https://mitsloan.mit.edu/teaching-resources-library/boeings-...

It's a super fun read. And you're absolutely right: Boeing caused the problem itself. Shifting blame to poor software "accidentally" produced by 9/hr engineers is not legitimate.

There was a bug. But it was only a problem because all the other redundancies and safeguards were shortcut and regulators and pilots were mislead about the severity of MCAS system's "corrections" and not given proper data to know when to override it.

Part of good engineering is having a solid understanding of your business domain and being able to spot things that just don't smell right.

An experienced professional software engineer on an industry standard salary, having been provided with nonsensical or missing requirements, should know when to impute and when to raise an exception. I'm not at all confident that a $9/hr engineer will do that.

Exactly. Seems just a bait article from someone who doesn't know shit about what happened (similar to Dave Farley video blaming the Cyberpunk launch fiasco on the lack of CI/CD).

You are underestimating how difficult it is to communicate with people from different cultures.

I did an electricity project with developers from the Ukraine. We spent a week being confused because the Ukrainiens expected the electricity bill to be calculated based on the size of your house and how many animals you have. The concept of a domestic electricity meter was completely foreign to them.

I also worked with a Ukrainian graphic designer who had no understanding of copyright law.

> Say some country in eastern Europe, where average salary is $500 / month. Then you only hire from the top CS programs in the country and pay them, say, 3 times the national average - so $1500 / month.

This is 10-15 years out of date. And even then it was hardly ever as straightforward as that. And only ever applied to junior and maybe mid level developers or those who couldn’t effectively communicate in English. High skilled one were significantly more mobile and were basically competing with a much more global pool of developers. They could relatively easily move (if not to the US then at least Western Europe so your potential savings were usually limited to the difference in CoL + some premium).

This led to pretty high inequality based on skill/experience e.g. to top CS graduates could expect their income to increase by 3x if not more over the next 5 years after graduation.

This assumes that

A) The institutions in the country make it possible to identify good from bad local candidates.

B) There are enough experienced people in the region to have a healthy functional workplace.

C) The outcome is at least proportional to the cost.

In my experience none of these are true.

Average wages for workers in Eastern Europe are way higher than $500/mo, never mind software engineers. Poland is $1,400, Hungary $1,200, Bulgaria $1,100, Romania $1,600, Lithuania $2,200, Estonia $2,000, Latvia $1,700. If you assume that software engineers will earn at least double the average wages of all workers, you're looking at $2,200-4,400 as a base. Paying 3x that to get the top people would mean $6,600-13,200/mo. Ultimately, you'd be paying $44-88/hour to execute your strategy.

Countries like Bulgaria and Hungary are both in the EU allowing workers to travel to Germany, France, etc. to get jobs. They aren't trapped behind immigration laws that don't allow them to move to places with higher salaries.

Even if you look at India, $10/hour wouldn't get you the best engineers. Mediocre engineers are earning that after a few years experience. Maybe you'll say that you'll pay triple that and that $30/hour is still cheap. Sure, $54,000/year is a cheap software engineer, but I think the best engineers have better options than being seen as a cost savings for some American company.

Frankly, it isn't necessarily about the money itself. It's about the attitude. If a company sees software as a cost center to be optimized/cut, their software will be crappy. If you're just focused on cutting costs, it's not just about how much you're paying engineers per hour. It's also about whether you let your engineers work on something that makes the product better, but might not be strictly necessary. For example, maybe there's a process the customer has to go through and many do it wrong because it's confusing. Will you let them spend the time to re-write it? Or will you say that it works so you don't want to spend money on that and customers will put up with it?

In theory, you could have more engineers and allow them to do more if they were cheaper. I've found that (in practice) when companies are looking to optimize salaries like that, they're also optimizing the time spent on things as well. They don't see their software as a strategic advantage to be invested in. They just see a cost center to be cut and that won't just be salaries, but also hours worked.

It also creates a crappy atmosphere where workers often don't care about getting things done beyond what they need to keep their job and not get yelled at. You're looking to work the system for yourself and minimize costs and they'll get a similar attitude where they try to work the system and do as little work for you as possible.

I'm not saying that you couldn't create a great company in Eastern Europe or India - they exist. But I think if your attitude is one of outsourcing a cost center, you won't get what you're looking for. Pay is relative, but it's higher than you think and you don't want to fall into the trap of just thinking that software is a cost center.

Plus, you don't even have to go that far to get cheaper engineers. You can pay half of what you pay Americans in France or Germany.

I don't know who this is so please name names.

What alleged company is this? (allegedly)

But also, lack of oversight, reviews, inadequate testing, etc.

I may not be $9 / hour but I still wouldn't trust myself to write aviation software. Nobody should have that kind of hubris, yet there's developers like that in all income tiers.

Trust no code, least of all your own.

But if you present your badly written requirements to a person who has the culture to ask questions, you at least have that safeguard. After all, it's a collaboration.

And when you work with an offshore team, quite often as contractors, you have the issue that you don't get top quality.

The in-between company will do its very best to hire low to maximize profit.

The good quality developers work either on local grown software, or leave the country for better opportunity.

And you are now working with a group of developers whose livelihood depends on saying 'yes' or not saying 'no.

And because of this you will not know whether there is an issue in the requirements until x time later and the plane goes down.

I really do not think the programmers are the responsible in this case. I'm not sure, but for what I understood, it was not a failure in the SW, but a failure to explain the pilots the changes in the behaviour of the plane.

Management at Boeing killed those 346 passengers.

If your bad code gets through to a literal airplane, then the entire QA Process is to blame, not just the software engineers/programmers.

> killed 189 people

$396 million is all that would cost. That is a fraction of the damage to shareholder value.

> I think that companies should rely on non-voting shares far more often, with a promise to not blindly follow profits over all else.

sure, but how is that relevant? it's not like the previous CEO of the company was sacked by shareholders because they demanded a 0.05% increase in net revenue.

> The more often capitalism strives to cost reduction and profit-maximizing at the cost of quality

Dumb capitalism only?

> Very limited options if I want to invest my money into an (engineering) quality-first company.

NVIDIA, Porsche, Tektronix...

Can you elaborate?

Why do you think $9 an hour is shameful? Doesn’t that entirely depend on the cost of living? $9 could be 3x the average wage in some parts of the world.

Done!