Boeing Engineers Lost Control of the Company
perell.com
perell.com
Was it really necessary to add the second part to explain why the merger occured? 'industry is consolidating' seems reason enough. The mixing of explanation with moralism in every sentence is grating. Put the moralism and ethics in it's own paragraph so we can understand the explanation first. If it's all mixed up, then I have to evaluate every claim in the article as if it might be ethical statements by the author instead of focusing on understanding first, and then considering the ethical outcome after having attained the understanding.
But it is still poor optics to rarely dog food your own products.
[1] https://qz.com/work/1169517/executives-are-dying-in-private-...
[2] https://fivethirtyeight.com/features/corporate-jets-commerci...
So it's really not a fair comparison and I wouldn't be surprised if they're functionally equivalent after taking into account all major variables.
But Boeing executives doesn't fly on these, since they're too large for even CEOs. Boeing executives fly on one of the three Boeing owned Bombardier Challenger 650s [1][2][3].
[0] https://en.wikipedia.org/wiki/Boeing_Business_Jet
[1] https://www.planelogger.com/Aircraft/Registration/N541BA/908...
[2] https://www.planelogger.com/Aircraft/Registration/N543BA/908...
[3] https://www.planelogger.com/Aircraft/Registration/N544BA/884...
I mean, do you really not understand how that sort of thing is done? They ask 100 people and then edit it down to the 5 most extreme responses.
I'm from Charleston, where the plant is located, & I have several friends that work at Boeing in assembly and quality inspection. What you see in the video is exactly the same sentiment I've heard for years.
I was really writing about work travel though.
That really is actually exactly what they did and are doing. When you don’t do tests because you are confident something will pass, that doesn’t change that you aren’t testing, and you are compromising quality on the basis of productivity. Here Boeing decided to go against the safe practice because that would have classified the plane as a different plane and incurred huge costs, but it would have increased safety and likely avoided the accidents.
They did compromise safety, they did so thinking that it wouldn’t matter because the overall safety would be good enough for no one to notice, but that gamble failed and others paid their lives to cover the bet.
If product A has passed testing and I made minor changes and made a product B if I can avoid running the whole battery of tests as if product B is brand new and not derivative then I totally will and Aerospace and Automotive operate like this all the time in every country in the world it's not poor practice.
Human factors design is nowhere near easy or settled science, not in iphone design nor in flight controls design.
Given.
The issue I keep coming back to is the flawed regard for human factors in MCAS. They paid enough attention to squeak by the artificial feel, but then completely ignored the fact that a pilot can't react or stay ahead of something on a plane that no one tells them even exists. Then when it became evident it might be a problem, they back pedaled, putting as much blame on the operators as they could.
They thought about it. They decided "Nah, they don't need to know." At some point the tech pilot even expressed concerns over the system, yet was still pressured to keep it out of the manual...why?
No one in good faith has anything to gain from that system's implementation details seeing the light of day. Unless of course, there was some other motivation, like avoiding the costs of compliance at some point.
The first of the 3 sets of pilots did this, and landed without incident.
The next set overrode MCAS successfully 25 times, inexplicably did not turn off the trim, did not override it the last time, and crashed.
The last set successfully countered MCAS twice, then turned off the trim. They did not follow the directions on Boeing's Flight Crew Operations Manual Bulletin: "Uncommanded Nose-Down Stabilizer Trim Due to Erroneous Angle Of Attack During Manual Flight Only", which the FAA turned into a mandatory emergency airworthiness directive.
-- Aviation Week, Aug 19, 2019
Extra pilot in cockpit. Not a CRM situation that's a given.
>The next set overrode MCAS successfully 25 times, inexplicably did not turn off the trim, did not override it the last time, and crashed.
They didn't know why the trim was kicking in in the first place, and the senior pilot was the one countering the trim. He handed it off to the junior pilot to sift through the manual, which ironically, would have been futile anyway as MCAS was dropped from the manual. The behavior did not present as a classic stabilizer runaway. Other pilots interviewed also pointed this out including in the Australian 60 minutes expose on the 737 MAX if I recall correctly, so I'm not sure that the same definition you're fond of is universally shared amongst pilots. Bad priming is also a human factor that is catastrophically lacking among other things not being taken into account.
Furthermore, if pilots were really held to account for knowing the hardware they're operating, they should have known the nature of the trim switch change between the 737 NG and MAX, and the consequences thereof; that wasn't included.
>The last set successfully countered MCAS twice, then turned off the trim. They did not follow the directions on Boeing's Flight Crew Operations Manual Bulletin: "Uncommanded Nose-Down Stabilizer Trim Due to Erroneous Angle Of Attack During Manual Flight Only", which the FAA turned into a mandatory emergency airworthiness directive.
They did follow it actually, but also notice the Note toward the end (see the appendix in the Ethiopian preliminary report. The line reads)
>Note: Initially higher control forces may be needed to overcome any stabilizer nose down trim already applied. Electric stabilizer trim can be used to neutralize control column pitch before moving the STAB TRIM CUTOUT switches to CUTOUT.
Now. There is something to be said for "Gee, did they read it", but considering the bloody thing includes a disclaimer that the AFM supersedes the bulletin due to not being approved by the FAA, the note being presented outside the main content of the directive, and the notification beeping issued in November, I don't have high hopes for the real effectiveness of the bulletin at clearly communicating the severity of the problem, something which simulator time would have made plain and obvious.
Then again, that's just me.
Unknown cause is symptomatic of runaway trim.
> The behavior did not present as a classic stabilizer runaway
How is runaway trim different from classic runaway trim? The trim was mysteriously turning on and trying to drive the nose into the ground. If that isn't runaway trim, I have no idea what runaway trim is.
BTW, I worked for 3 years on the Boeing 757 stabilizer trim gearbox. I know what runaway trim is. The cutoff switches are there as the last resort to stopping it, and they're prominently and conveniently on the center console for darn good reason.
> if pilots were really held to account for knowing the hardware they're operating
The point being, even if they knew what MCAS was, the corrective action is the exact same thing as in the instructions provided by Boeing, before and after the Lion Air crash.
> They did follow it actually
I'm glad you read it, but read it again. They did not follow it.
> clearly communicating the severity of the problem
Considering that it was an emergency mandatory notification, and the LA pilots had already crashed, I don't know how to harder communicate the severity. I'm not a pilot, but if I was a MAX pilot, I'd certainly not want to die the same way the LA pilots did, and I'd pay attention to an AD explaining how not to die.
But I do agree that evidently pilots needed more training to understand what to do about runaway trim, which they need to understand regardless of whether MCAS is causing the runaway trim or some other problem. There's a reason the runaway trim procedure is something that is a memory item, meaning pilots are supposed to know what to do about it without needing to consult the checklist. Training in this is clearly insufficient.
The second part is key to me. Inadequate system safety analysis seemed to give them the green light in decisions to save money (or make more money with safety critical 'options'). It's not like they knew MCAS would have an out-sized chance of downing a plane. My guess is they didn't know what they didn't know and it led them to have overconfidence in their ability to properly gauge business/safety trade-offs
To play devil's advocate, this isn't a moral statement. The author is saying this is true based on actual events.
Is that not pursuing profit at the expense of human lives?
The fact that the system safety analysis didn't label it 'catastrophic' seems to indicate they didn't know this. This doesn't absolve them of the responsibility that they should have, or that they still didn't follow their design procedures stating redundant senaors
The fact the system safety analysis didn't call out an MCAS failure as catastrophic seems to indicate they misunderstood the system risk which is different than claiming a catastrophic failure is an acceptable risk
They have to acknowledge at some point that lives may be at risk in their business. Whether or not that risk level is a acceptable and that they properly gauge that risk 8s another matter
You can tell how seriously they took human lives by their default reaction a plane crash. It's always "pilot error": https://www.bbc.co.uk/news/extra/sd9LGK2S9m/battle_over_blam...
So yes, if you fall asleep or don't have your hands on the wheel while driving, it is your fault for being a shitty driver. I hate how Tesla has marketed Autopilot, but I cannot feel bad for the people stupid enough to fall asleep at the wheel.
Matter of fact, I don’t even care how your company is subdivided.
Could be a fiefdom for all I care.
But if you sell me something saying it does “X” I bloody expect “X” to mean “X” and not “not X”.
But that just me, I’m old fashioned that way.
Hell, both Boeing and Tesla can be said to have indulged in the same sin: not accurately communicating the nature and capabilities of their product to the end user
Taking the viewpoint that caveat emptor absolves the manufacturer of any complicity in mass producing and delivering for use dangerous systems without clearly communicating the underlying implementation details to a reasonable operator is just beyond naieve to the point of asking bad actors to take these types of risks in the least responsible way possible. In fact, the only reason Boeing is getting slammed so hard and Tesla isn't is entirely due to extra regulatory burden that aircraft manufacturers have to deal with that automotive don't.
Now, I personally take the onus of understanding my vehicle seriously, but not everyone does, and nevertheless, they still must be able to drive. Therefore I personally hold manufacturer's responsible for being good corporate citizens and not indulging in deceptive practices.
Not that that gets me or anyone else anywhere. It's the principle of the thing though.
Because we're talking about legality, not morality. I have said in my original comment I hate how Tesla markets this, but that does not absolve the driver of responsibility for piloting their multi-ton metal box at high velocity on public roads. What you're effectively saying is you want some entity, likely the government, to hold Tesla accountable for their claims. Except you will never find an official statement from Tesla saying you can drive without paying attention (there are numerous unofficial statements, and even a video of Elon Musk personally demonstrating the technology without paying attention on national TV) - so it's a question of proving fraud at that point. And proving fraud is an insanely difficult process, because you have to prove intent. Even proving neglect in this instance is challenging. I do wish there were more stringent regulations about advertising for driver-assist systems, but currently they do not exist.
Yes, in a perfect world, I would love for Tesla to not get away with this shit. But rather than only blame Tesla, perhaps we should recognize that driving is a privilege, not a right. You have to pass a test to get your driver's license. You are responsible for the car so long as you are driving. If you are so easily duped that you can be dissuaded otherwise, you are likely not fit to be a safe driver.
Ultimately what I care about is how safe the roads are. I care much less about pointing the finger. Tesla deserves criticism for their disingenuous marketing, but so do lazy drivers who literally fall asleep at the wheel without even checking to see if that's okay.
I think it's becoming clear that at least Boeing have also indulged in "actively deceiving the regulator" as well as the end user.
(It may be that Tesla have also done that, but at least so far nothing like as egregiously as Boeing's recently revealed internal comms...)
I'm curious, because what I've read as a layman led me to believe Boeing legitimately didn't know how dangerous it was and their processes/culture helped undermine facilitating a better understanding. Still not good, but a different world than actively deceiving regulators. It almost seems like their cognitive biases deceived themselves
Onus does not apply to the safety of consumer products. For example, it is the responsibility of an automobile owner to drive a car with brakes in good working condition. Period.
But it is also the responsibility of an automobile manufacturer to issue a recall and fix faulty brakes in a car they sell.
It’s not an either-or. We don’t say, “Because the onus is on the driver to drive a safe vehicle, there is no fault to be laid at the manufacturer’s door. Caveat emptor.” We hold both parties responsible to some degree or other.
In sum, yes, drivers are supposed to know the limitations of their equipment and operate it responsibly. But also yes, manufacturers have responsibilities around the safety of the hardware and software they sell to people to operate as motor vehicles.
The driver’s responsibility does not absolve the manufacturer of its responsibilities.
But I think the solution to the problem describe upthread is more along the lines of "we need higher standards of truth in advertising".
While I agree that "the onus is on the driver to understand how their car works" - I think it's entirely understandable that drivers base that understanding on what the sales guy and marketing team told them when they researched and bought it, and that if they use works like "self driving" and "autonomous" and "autopilot" that it's reasonable for drivers to interpret them in their commonly understood meanings. Disclaimers in legal documentation redefining commonly understood English words are _not_ OK in my opinion.
And when it's the marketing department saying "X" in big print in flashy brochures and ads, and the legal department and their small print saying "not X in ways that kill purchasers expecting X", I'd expect company officers to be facing jail time.
I'm old fashioned that way too.
At the fundamental level, Engineering is a social/ethical process, not a technical one. Yes, engineers tend to have technical skills but when you use those skills to design a bridge you must always keep society (the users of your product) at the forefront of your mind. What separates engineers from "skilled people" is professional ethics. An engineer (structural, software, aerospace, chemical or whatever) who does not consider the ethics of their work cannot credibly claim to be an engineer at all.
To "put the moralism and ethics in its own paragraph" is to sequester the ethical considerations somewhere where they can be conveniently ignored in service of the cult of CEO worship.
edit: Instead of anonymous downvotes, would anyone care to let me know why they find this comment so inappropriate for HN?
Right now I feel like there is a lot of hostility aimed at the "ethically active engineer". I've seen time and time again people pushed to back down because they're "rocking the boat."
On the other hand, I look at stuff going on in the Free Software movement where cancel culture seems to be being weaponized (by whom, and to whose ultimate benefit I can't really tell)as well.
There is a balance, but I really do agree that if you aren't actively engaged with the ethics of the process you're facilitating, you can't really call yourself an Engineer. I have lists of projects I won't do, and I have fewer qualms with letting an employer know they're crossing them as I get older.
Unfortunately for me the point is a little moot, as there is no PE for Software fellas anyway, and I'm a few years away from being able to work my way educationwise into a field that has a program for it.
Just wanted to let you know you aren't alone in your sentiment.
It is just wrong.
>Engineering is a social/ethical process, not a technical one
This is just bizarre.
>Yes, engineers tend to have technical skills
...tend to have???
>What separates engineers from "skilled people" is professional ethics
I'd argue that it is specialized education
> when you use those skills to design a bridge >An engineer... who does not consider the ethics of their work cannot credibly claim to be an engineer at all.
A bridge is usually designed by a team of engineers, who will get a specification (usually from another team of engineers and other professionals).
One engineer may be a geotechnical expert, another might design piling, a third will specify the cement and aggregates in the concrete etc etc. Each contribution must meet the applicable technical standards and specifications. Apart from making sure your work is correct, how do you bring ethics into calculating the amount of rebar in your concrete slab??
I don't know about you, but there are lots of people I know who call themselves engineers but aren't actually very skilled at all.
>Apart from making sure your work is correct, how do you bring ethics into calculating the amount of rebar in your concrete slab??
I think it's pretty obvious that I was using "design a bridge" as a proxy for any engineering work. But to answer your question: by making sure you do your due diligence with your work, and push back against superiors who want you to cut corners in your analysis for money, of course.
The same way it works in any discipline of engineering. The engineers at Boeing appear to have been unsuccessful in changing the cost-cutting culture of their company, and the executives who ordered the cost cutting are ethically deficient. To put the ethics aside is to absolve the executives of responsibility. If you want a technical writeup of the MCAS debacle, the issue has been discussed ad nauseam in the crash reports. But those reports will lead to no material change if the CEO and other execs aren't willing to change their expectations.
Part of which in many programs includes familiarization with ethics coincidentally.
>A bridge is usually designed by a team of engineers... how do you bring ethics into calculating the amount of rebar in your concrete slab??
You are not a calculator. You are responsible for ensuring your specific part is correct, yes; but being conferred with the power to halt something that isn't up to snuff or to reject a plan for unsoundness also implies a responsibility to exercise that discretion. You (as the engineer) have those technical chops that others do not.
If you bow out and rubber stamp without a thought to the consequences of your actions, you aren't fulfilling your duty to the Public. With a bridge it's more straightforward, so it doesn't necessarily avail the civil engineer of the opportunity to exercise that discretion, but it is nevertheless there.
Even in bridge building, the balancing of human civilization's advantages against its environmental cost is a matter of public policy. Engineers may be consulted to assess environmental impact or to provide lower-impact options, but "should we have a bridge here or not" is ultimately a question for political leaders, not engineers.
I neither prescribed a particular ethical conclusion nor required consultation for my definition. There are lots of people who choose not to work in Defense because of the ethical considerations. There are others whose personal senses of ethics are not offended by the idea of making weapons, or at least are assuaged by the money. I would argue the same interplay between ethics and compensation is at work in e.g. Facebook and other surveillance firms. Quite obviously some people don't mind working for Facebook, even though I personally would find that repugnant.
What I'm not saying is that someone who works for Facebook "isn't an engineer" because I happen to disagree with their ethical choices. I'm saying that if you call yourself an engineer and you don't consider/ignore the ethical consequences of your work then you are an impostor.
The link to the topic at hand/reason I brought this up is that the 737 MAX issue (or the MCAS issue) isn't a technological problem so much as an ethical problem. Considering the technical faults in isolation ("putting the moralism and ethics in their own paragraph") is deliberately blinding one's self to the crux of the issue.
The root cause, the reasons why it doesn't do what it's specified to do stem from an ethical problem.
When you are in senior management, you make decisions based on their overall effect on the company's financial health or future growth. That is your job, just like it is the engineer's job to make decisions based on overall safety and reliability or future capability.
The debate, or perhaps the proof, is what happens next. Every decision, engineering or management, needs to be evaluated post implementation to evaluate its effectiveness. Are we better off financially? Are we more capable? Is this system more reliable? Is this system easier to change while continuing to validate it is meeting its design goals?
Both management and engineering resist having their decisions "colored" by their negative impact from the "other" group. Engineers don't like it when their clearly correct decision is overridden by management's feeling it doesn't make "business" sense (which is usually code for it reduces margins unacceptably). And management doesn't like it when their decision is negated by engineering telling them it will cause their products to become unacceptably unreliable or unsafe.
In a properly functioning company it is the role of executive management to combine both the needs of engineering and the needs of the business into an effective outcome that everyone can live with and keeps the company alive.
Articles on the acquisition suggest that Boeing acquired MD both because the industry was consolidating and because MD was making more money per plane than Boeing was. They wanted both the customer base of MD and the "DNA" or managerial understanding of how to build planes that were more profitable. Here in hindsight the implication is that the additional profit margin came from skimping on safety and reliability engineering.
At some point, there was no doubt a review on the 737-MAX, where the engineering teams raised concerns about MCAS and the impact on safety and reliability. And the costs associated with addressing those concerns were also discussed, both cost to implement and time to market costs. The result of that review was to pursue certification on their plan of record and to not incur the additional certification and time to market costs that engineering advocated were needed.
That decision was explicitly the pursuit of profit margin at the expense of product safety and reliability. As human lives are transitively related to safety and reliability of the aircraft, it is a reasonable statement to make about pursuing profit at the expense of human lives. That is what the equations reduce to.
I find this statement very troubling. You seem to be absolving the corporate/C-level class of any moral or ethical culpability when it comes to the consequences of their decisions, so long as "it makes business sense".
[1] https://en.m.wikipedia.org/wiki/Triangle_Shirtwaist_Factory_...
It should be possible to screw workers without endangering lives or the bottom line. Boeing did that long time. So did Airbus, GE Aviation, Rolls-Royce, Bombardier.
Moving it to a later part of the text is a great way to get people to ignore it.
Boeing started said consolidation itself by having McDD buy it Boeing with Boeing's money.
The reality is that trend #2 is orders of magnitude larger. Any reversal of that by ‘AI’ would be the biggest jobs boom in decades for western nations, and according to this article, would increase quality and therefore trade balance as well.
The other reality is that manufacturing automation with ‘AI’ is hard, takes a lot of human effort, and is a huge part of a company’s value (e.g. Tesla). Outsourcing is often selling off this value at a negative price. It is done because Wall St analysts don’t know how to price this asset.
Like it or love it, it is investment analysts that determine company culture. The good thing is that they are greedy; they may be misinformed, but they are predictable. Everything above liquidation value is held in the heads of the employees of the company. Its actually not that hard to compute, and if business schools taught this, corporate behavior would change.
Anywho, the thing that was interesting to me was how much Toyota and BMW were willing to invest in what is more or less an engineer's bootcamp- like baseball teams playing out-of-league teams in practice to get better- and that that was seen as a worthy investment by the engineering team's superiors.
Some companies still value engineers. That's where I want to be.
It's uncomfortable to think about but a management fashion that silicon valley helped promulgate killed those 600 people.
Now finally that China starts being too expensive, once investors will realize it is complicated and costly to outsource to Vietnam, India or African countries, I hope this effort will resume.
I wish Mr. Stoller took more time on this:
> The Federal Aviation Administration, having outsourced much of its own regulatory capacity to Boeing, didn’t know what was going on, and Boeing didn’t tell airlines and pilots about the new and crucial safety procedures.
This seems like big deal to me.
The gist of it is extremely simple: most fatal accidents happen because of a sum of mistakes, a chain of more or less important errors which, taken individually, would seldom lead to an accident, but together form "the perfect storm". This is btw the true meaning of Murphy's famous quote (especially the 'worst possible time' part, e.g. software failing in-flight as opposed to during tests), and how we avoid such outcomes.
The beauty of accidentology, beyond engineering / repairing honesty, beyond tons of carefully written procedures and double checklists, is also quite simple: to systematically track the origin of mistakes, to reward people who were involved in finding the issue, and fix the problem once and for all through better practice, better documentation, changing procedures.
It contributed to make air travel statistically the safest means of transportation for human beings (the rationale was that plane crashes were so traumatising to the general public that it wasn't enough to be "better" than cars or trains or ferrys; air travel had to be 99.99...% safe to succeed).
In this case, we indeed have multiple failures which form a chain of events: the manufacturer (Boeing) cutting corners on safety, the regulator (FAA) also cutting corners in what looks like plain and simple corruption here, and somehow everyone else (companies, pilots) taking for granted what Boeing/FAA said. All in the name of more profit, for a ridiculously tiny part of the population. The results are the tragedies we witnessed. And while the first crash should have been a wake-up call, the fact that Boeing themselves didn't come forward and did let a second crash occur is borderline criminal (for courts to decide, anyway, but the question seems salient in this case).
It's all extremely worrying and sets a dire precedent for the USA as a lax or incompetent regulator, and a champion of big corporations now apparently digging way too far down the 'profit hole' — it has literally become a grave by that point.
> The key decision was, rather than fix the fundamental aerodynamic control problems caused by the new engine, to bandaid the existing 737 software, while pretending that flying the 737 Max was just like flying old ones. That way, airlines would be able to buy the plane and not have to retrain their pilots, as pilots must be re-certified any changed flight procedures but don’t have to be recertified for new models with unchanged flying qualities. Unfortunately, the aerodynamics of the 737 body didn’t fit with the Max’s bulkier engine, which was obvious during the first wind tunnel tests.
This is the core point which the author seems to be trying to make, that outsourcing necessarily raised costs because it's a de facto defective strategy for improving margins. What the author fails to delve into is precisely why subcontracting failed to deliver as imagined - did subcontractors fail to deliver airplane parts within tolerances? Were the specifications provided to subcontractors faulty? Did the outsourcing contracts fail to require compensation for failing to deliver on-time and within tolerances; did such requirements exist but in reality such financial obligations caused subcontractors to run out of money / go bankrupt and default on their obligations? Was no insurance purchased to hedge against these risks, so that costs would not explode along with delays?
Part of smile theory is that when you decide to outsource lower-margin operations, the ability to outsource effectively in and of itself becomes part of your core competency. That Boeing executives failed to reduce costs as planned through outsourcing is less proof that outsourcing is inherently poor practice and more proof that competency at shifting numbers around on bean-counting accounting sheets is not the same thing as competency at outsourcing.
What I'm arguing is that, in order to effectively outsource, one must get good at the so-referred training and integration costs, while hedging against logistics issues. Otherwise you will get, as you prophesy, an inferior product. But I'm not convinced that's inevitable.
Of note, keeping everything in-house can result in its own logistics issues, as the organization becomes necessarily larger and must pay related overhead costs. So deciding to keep processes in-house doesn't mean you get to completely ignore "logistics loss" because you're paying a similar price in its stead.
Are there well known examples of companies starting as a low to mid-tier player in their sector, and gaining market share through the use of outsourcing?
I think by now we have demonstrated that this is false, no? In the 1960s, it was a popular idea (especially among managers) to assume that a company can have a core competency of "management" rather than making products or services. If true, such a core competency would mean
1) You can take a great manager at an Ice Cream making company and move him over to an Aerospace company and he'd still be a great manager.
2) Rather than being facilitators who helped to coordinate talented people, managers had their own talent at "managing" and so you can pay them the kind of superstar wages that were previously paid to superstar performers.
3) The above two skills could be efficiently measured and identified to allow scaling them to a large organization.
This idea was obviously very popular, and so a business structure called the "conglomerate" arose where one huge company would buy and own many different smaller companies all with completely different product lines, because the competitive advantage of the conglomerate was viewed to be "managing" rather than producing, and thus the superior managers would improve margins at pretty much anything they were put in charge of.
But unfortunately what ended up happening was that the conglomerates underperformed. There are still some conglomerate hold outs, like Berkshire Hathaway, but everyone knows that this is because of Buffet's investment acumen, not because Berkshire has a stable of excellent managers. Perhaps there is such a thing as pure "management" skill, rather than say, "Engineering Management", but if such a skill exists, it's sufficiently difficult to measure and/or replicate from one product area to another that none of the businesses who thought they had this skill as a core competency actually had it. This remains the case today -- there are many well managed companies, and many talented managers, but 'management' has never been and cannot be a core competency of any firm.
So this idea that Boeing can shed its core competency from building planes and replace it with a "coordination" core competency is a rehash of the conglomerate dream with the same disastrous outcomes, and many people who saw what Boeing was doing knew exactly what this would lead to.
No, that isn't the argument at all. A manager at an ice cream factory doesn't need to understand aircraft manufacturing tolerances. If a manager at an ice cream factory outsources the production of chocolate, and the chocolate that arrives as an input is slightly off-grade, it's not going to cause thousands of tons of aluminum and human flesh to come crashing out of the sky.
Risk management has its similarities across disciplines, just as the principles of Lean Manufacturing are similar across disciplines. But you can't effectively outsource something you don't intimately understand, which (by the way) is why it's impossible to outsource research.
Such a talent is not a core competency of any firm.
More importantly, such a manufacturing process is incompatible with innovation, where you have to be able to make lots of mistakes as you stumble towards a better design iteratively. It only works when everything is known to spec before the contracts are signed -- e.g. when you are not innovating.
This type of design agility can't be extracted from the minds of conscientious engineers working together on a shared vision, and parceled out to for profit contractors looking to do spec work.
Similarly risk management certainly adopts a common language, but the idea that you can actually manage risk in a quantitative way is not something borne out by experience. Rather, you can cover your a$$ and say you have sign off from you GRC people so that when stuff breaks down you can point the finger at someone. We just exited a massive financial crisis, and we see security breaches on a massive scale, most of which happen in firms with well funded GRC teams. Again, wishing that you can control these variables does not make it so. Putting together a bureaucracy to control something doesn't mean that it's controlled.
No it's not, which is why agile processes have less risk than waterfall processes. You're not making the argument that Boeing should keep all of their manufacturing in-house, you're explaining why Shenzhen has become the hardware manufacturing capital of the world. Shenzhen isn't controlled by a single company, and the ability to quickly order parts from other manufacturers in the Shenzhen area allows hardware manufacturers to respond with the agility you describe. Who controls the individual manufacturing processes is less important than the speed of the feedback loops.
> the idea that you can actually manage risk in a quantitative way is not something borne out by experience
Well, Wall Street would disagree with you.
> We just exited a massive financial crisis
2008 happened not because the underlying risk models were wrong but because people were sold high-risk instruments on the fraudulent premise that they had almost no risk at all, and the rating agencies which mislabeled the risk hid behind the First Amendment. If the high-risk instruments were honestly represented at sale, they wouldn't have been purchased by people seeking low-risk instruments, and the portfolios of most market players would've been well-diversified as planned and able to withstand those kinds of shocks. That the rating agencies were able to hide behind the First Amendment (risk assessments are not political speech) and that nobody went to jail for the fraudulent risk assessments remains a great injustice, in my opinion.
> we see security breaches on a massive scale
Because law and regulation do not set sufficient financial penalty for insufficient security practice and the market at large has shown not to care enough about security to punish players with poor security practice, so the financial risk of a breach is (in my opinion, unacceptably) too low to cause a change in corporate behavior. Again, not an issue with risk management, just people responding to systems as designed.
> Putting together a bureaucracy to control something doesn't mean that it's controlled.
Sadly I actually agree with you on this, which is why I wrote, and I quote, "competency at shifting numbers around on bean-counting accounting sheets is not the same thing as competency at outsourcing".
When the smooth talking suits show up and tell their pretty dictions, there aren’t enough people who know how to push back.
But where would you find these classically trained managers who haven’t drunk the Taylor koolaid?
The product plummeted soon after, product market fit got lost as the direction became more and more political and by the end, the ones that came out on top and ready to move to their next venture were the managers that managed to tick their measurements and KPIs even though they were totally disconnected from the operational space.
But, they do wield weapons that us engineers often don’t have the time not desire to sharpen...
Right, this is something more engineers need to get better at, this is nothing more than first year law school debating which is teachable. Seems like Management training for engineers should include this skill for good. Also having a backbone to fight too.
https://news.ycombinator.com/submitted?id=davidperell
Why bother to read this exactly? There is always a question of what you read being true and sure it's fun to poke positive or negative. But honestly having a flashy website or blog is not enough.
(Kudos David for being opportunistic (I really mean that btw) pulling this off).
Furthermore, the author also gets a number of things wrong about his research, and also presents statements as pure fact which are purely false.
Here are three examples I pulled out—in order from minor to argument-disqualifying.
1. The author cites innovation on the 787 as having electrically operated systems in place of hydraulics. He meant to say pneumatics. The 787 does not have an engine air-bleed system, using solid state electric systems to power the aforementioned components instead of the system of valves and ducts which manage the flow of high pressure air coming off the engine, as is the norm on other aircraft.
2. The author sites engines being outsourced historically because of a natural difference in engineering expertise between engine and airframe. While engine and airframe engineering are different domains, the real reason airframe manufacturers don’t make engines and vise-versa is entirely due to the Air Mail Act of 1934. I suggest the OP research the United Aircraft and Transport Corporation.
3. The OP’s central argument of cost control, and particularly outsourcing, as being a central driver to engineering problems with the MAX is hindered when he cites example and supporting arguments primarily about the 787 program.
Yes, cost control is an overarching issue and one that is being investigated vigorously, and 787’s level of airframe outsourcing has been a key problem to that particular program’s profitability.
However, the author fails to understand that the outsourcing he talks about is made at the program level. Citing the 787 as being representative of Boeing as a whole is spurious. The author also fails to understand how outsourcing effects engineering both at Boeing and down the supply chain.
In truth, the level of outsourcing on the MAX is largely no different than on the NG. (The previous model 737 that has been flying safely for decades, which went into service 2 years before the MD-Boeing merger.) It has been common since the advent of the jet age for there to be separate suppliers for engines and components—i.e landing gear or avionics. This does not mean innovation and engineering expertise is past over the fence—-airplanes are in fact built on strong partnerships between firms, and that includes partnerships between engineers. This has always been how the industry works.
Now, tying this back to my 3rd example: The 787 supply chain problems the OP sites are in regards to final assembly of fuselage and wing structures — this is a specific area in which one could argue the 787 went overboard in outsourcing. This, of course, has nothing to do with MCAS—a piece of flight control software—nor the AOA vanes that feed it. Flight control systems, sensors, and vanes are typically outsourced, as they are on the 777, 767, 777, and 747 (all of which engineered and built before the merger.)
...
If we really want to add value to the conversation about Boeing’s culture, the 737 MAX, and engineering aircraft—then let’s please build some knowledge first about how aircraft systems are actually engineered.
Disclaiming that nothing can be definitively said about the MCAS crashes until all the investigations are complete and known, I strongly believe the issue is not a software failure but a systems failure, and particularly, a failure in the management of the safety classification and assessment process.
There is also the matter that the 737 is a very old program. Beyond the argument that it was a mistake to try to modernize a 1967 design and architecture a 3rd time, there is also the problem that the 737MAX doesn’t necessarily adhere to current engineering standards for safety assessment—especially at the systems level. This is because the 737MAX was engineered under the authority of the original 737 certification.
For aerospace engineers working in industry, it would be useful to map what we know about MCAS and the 737MAX against the requirements outlined in DO-178C and, especially, ARP-4754. Unfortunately, the 737MAX doesn’t subscribe fully to ARP-4754 methodology, since it’s a continuation of the program certified in 1967 and not a new type.
It seems like a systems & human factors engineering failure. Not a software failure.
Why deliberately untrained? Because the whole selling point of the design was that you wouldn't have to retrain your pilots because it would handle just like the old plane.
We're also straining the definition of "failure" here as it relates to software. Software doesn't really fail in the way considered in the realm of traditional functional-safety. It doesn't wear out like a cable or a capacitor. But, I digress.
It's simply crazy. The only plausible explanation is that Boeing did not perceive MCAS to be safety critical, even though it had control of the aircraft, and so they didn't bother to apply any redundancy to it. Boeing seemed to wrongly assume that any failure modes would appear as a runaway trim.
That's commonly said about the Apple-NeXT acquisition too — NeXT paid a negative $429 million to take over Apple.
Obviously the results were much different there...
> That's commonly said about the Apple-NeXT acquisition too — NeXT paid a negative $429 million to take over Apple. (...)
I guess that is why it is considered a joke. It sounds like the management teams of NeXT / McDonnell Douglas benefited much more (in the long term) than the investors of their defunct, purchased companies. Benefiting through greater compensation and clout without really taking over (i.e. owning) the purchasor's company.
- at least two executives including the CEO should get life in prison
- at least two people from the FAA should join them
- Boeing should be dismantled and become at least two separate companies.
- The FAA should also be rebuilt from the ground up.
- Create a new type of corporate tax that helps ensure the FAA has a more adequate budget.
- New law that ensures that CEO and some other upper-level decision makers need to have senior engineering experience. (Ideally there would be no executives at all, just engineers, but that's too unrealistic even for my fantasy here).
Can someone explain to me how this is possible? In my understanding If you move the engines forward, it would move the center of gravity forward as well, and this would push the tail up and pull the nose down - exactly the opposite from what's written in the article. Does some fancy aerodynamic effect play some role here that leads to stalling?
What isn't is the extra lift at high-AoA and longer lever arm for the lift force to operate through. MCAS was meant to counter that, and arguably would have been a perfectly reasonable fix if they'd have designed it to the proper degree of redundancy, and actually told pilots how it worked and that it existed.
I believe the issue is precisely this - the engines were moved (and they are bigger so more powerful perhaps?) forward and the thrust probably pushes the nose up.
Boat might be a better model, or even better, use your hand hanging out the window of a moving car.
If you taped a couple of projections under your hand, and slowly increase the Angle of Attack of your hand to get a feel for what the old design would do.
Then do the same thing but with those protrusions up in front of your hand.
You should notice that it becomes harder in the second arrangement to counter the tendency of your hand to continue pitching up after a point. Assuming your weird mini-not turbine construction stays together in the airstream.
If you move CofG forward, and up you move the balance point forward and up, reduce elevator authority, and with similar tail mass now a metre or two further behind the balance point, it will want to sit tail-heavy. So they tried to trim it out, and we are where we are.
Making the engine more powerful (as is the case here) means there will likely be more pitching up moment (turning about the aerodynamic centre or AC) due to the position of the engine relative to the AC. Speaking of which, there is an AC, an aerodynamic centre that is separate from the center of gravity. It represents the point about which the plane pitchs up/down when a force is exerted on it.
Not only is the engine more powerful, but the different position it was placed in (forward primarily) would also create a different force about the AC, and change the forces the other parts apply to the plane (by moving the AC and CG).
There might have been more to it but even a few months after really diving into this the details are getting fuzzy... plane stability dynamics can vary greatly.
Think of it like attaching an extra conditional lifting surface to the wing. In normal, low AoA flight, no pressure differential is created. At high AoA though, that extra lift works through the lever arm of the nacelle assembly.
Control feelwise, this would result in a slackening of control force needed to yield more degrees AoAby the plane after a certain point. That's a regulatory no-no.
For whatever reason, the basic facts of this whole debacle are consistently wrong in reporting and discussion. Common falsehoods:
* The 737MAX is inherently unstable. No, it is a stable aircraft as required by FAA regulation that all passenger aircraft be as such. Nothing about the airframe or design is even that remarkable, beyond the simple fact that it is different from the older models.
* MCAS is anti-stall. No, MCAS does exactly what it's named for, to modify and augment the 'feel' of the flight stick for the pilot. Because Boeing uses a 'pilot in the loop' design approach, the physical feedback from the flight stick is critical to safe flying, and cannot be faked.
What MCAS does is make is to that you have to apply a proportional amount of force on the flight stick to pitch up a given amount (regulations define how far from linear this relationship can be). With the new engine placement, it would not require as much force as you would expect from earlier planes, which could lead to pilots approaching stall conditions unknowingly without that feedback. There are still anti-stall measures to prevent an actual stall (e.g. stick shakers), but the intuitive 'feel' of flying would be lost.
The initial MCAS design was reasonable, but test pilots felt that it wasn't strong enough, so it was changed. It is at this point, with no critical review being triggered, that the failure took place. The stronger system put pilots in a catastrophic situation in the event of a malfunction, and two independent flight crews failed to overcome the error.
It’s easy to look back with retrospect and make sweeping judgements about the entire company and the FAA.
But fundamentally this was all over, as the author mentions, a single software bug and a lack of backup sensors to offset the risk.
In addition it was the technical pilots who asked for the stick to be harder than it was, which added to the difficulty in that unique situation.
The MAX was otherwise a very safe plane was it not?
There will always be limits to how effective regulatory oversight can be.
In software this hack is explicit, so perhaps Boeing engineers can be forgiven for not seeing it when it straddles hardware and software.
I'm hoping we get more details so we can help build a text-book case against software outsourcing.
"Boeing considered MCAS part of the flight control system, and elected to not describe it in the flight manual or in training materials."
It... "could not be overridden by pilot resistance against the control wheel as on previous versions of the 737."
"Following the Lion Air accident, Boeing issued an Operations Manual Bulletin (OMB)[13] which outlined the many indications and effects resulting from erroneous AOA data and provided instructions to turn off the motorized trim system for the remainder of the flight, and trim manually instead. Until Boeing supplemented the manuals[14] and training, pilots were unaware of the existence of MCAS due to its omission from the crew manual and no coverage in training.[12]"
It wasn't just "a software bug". It wasn't just culture cutting corners. Even after the first crash they continued to cover up that MCAS was the cause. After the second crash they still refused and it wasn't until airlines started grounding planes that they caved in.
There's a lot more than regulatory oversight could have done. We haven't reached the limits in the slightest.
Its that politics got in the way of doing the right thing, which is declare the airplane "new" and re-train the pilots (vs a plain modification). Boeing didn't want to do that as customers didn't want the retraining expense so they used lobbying and cut the corner.
(Aircraft doors are interesting on their own as well, due to the assumptions that drive design, testing, and certification.)
Not unreasonable, doing stab trim cutout is a memory item for most operating manuals - pilots were already the circuit breaker for other stab trim issues.
Automation working - great. Maintenance failures? Provide error messages (which occured here). Maintenance issues not resolved and automation misbehevior? Stab trim cutout memory item.
Despite all the claims that the stab trim cutout did not work all real evidence is that it did. Other issues around pressure on flight surfaces not explored properly making recovery in overspeed and other conditions very hard.
Badly designed system? Definitely. Brittle when facing poor maintenance? Clearly! Not taking into account broad pilot population. Of course. Definitely should be fixed, and frankly automation probably will be increased to decrease assumptions around pilot behavior which can be variable.
This accident was the result not only of this bug but a chain of issues. That's why flying in the US is pretty safe, the chain from procurement to maintenance to pilot training is strong so compensates for poor elements.
If you follow SAA technical for example there is evidence overseas that bogus parts are making it into plane repairs, with predictably bad results. We are not seeing that in the US (yet), though if inspections get soft I'm sure it would show up here too. Those bogus repairs will mean that some automation will break and the plane may not behave properly in ways that could have been designed around with better assumptions.
My prediction? We are going to get iphone style authentication / serial number chips in key plane parts with electrical connections and the plane will eventually refuse to fly without the right parts - just like your coffee maker and iphone. Right now it is trust based.
As you'll find in this article below, Boeing specifically removed the yoke-back function, which is the natural pilot input to nose up the plane. They also specifically for the MAX removed the separate switch for automatic stab trim assistance, so you have to turn off the motorized function entirely to turn off automated inputs including MCAS. The Lion Air crew could be heard on the radio saying that they did disable the stab trim, but in this situation the force required on the manual trim wheel was beyond the pilots' strength. The pilots turned the electronic stab trim back on in hopes of using the powered system to get back up, after which the undocumented and unstoppable MCAS nosed down the plane again, killing them and their passengers.
https://www.seattletimes.com/business/boeing-aerospace/boein...
Your post reads like a sophisticated attempt to blame the victims, or at least the pilots and ground crews of the victims. These crashes were not caused by poor pilots, but rather secrets on top of secrets, all by Boeing, all for money.
The MAX had new aerodynamic characteristics, but that needed to be a secret.
So MCAS was added to the electronic trim assistance to hide the new characteristics, and the switch to disable it was removed to protect the secret.
A faulty AOA sensor was now able to bring down the plane, but in order to keep the secret the single AOA option was still available and sold. Maintenance crews regardless of quality would not have been aware of the criticality of the AOA's secret role as a single point of failure for the whole plane.
When the first one crashed, secrets covered it up.
When the second one crashed, it still took all of those "ppor maintenance", "broad quality" non-US airlines and regulators to bring a stop to it.
And apparently the secret-keepers are still trying to hide.
Please don't blame dead pilots for this.
https://www.seattletimes.com/business/boeing-aerospace/newly...
Another regulatory no-no that wasn't brought up until the certified aircraft had been out in the wild long enough to kill people.
First -> if this is your worst failure mode (ie, worse than pilot error, mechanical fatigue and failure, engine failures, fueling issues, crazy passengers etc) then you have a plane and overall safety system that is incredible.
Second -> this wild set of events again does not crash the plane - though it would put the pilots in a bad spot similar to the existing MCAS issues.
Or would be, if it didn't violate design constraints that demand no single point of failure can have the capability to result in loss of the aircraft. I.e. components with failure profiles of catastrophic severity MUST have redundancy. You test the least likely events to ensure you have sufficient fault tolerance for that very reason.
>Second -> this wild set of events again does not crash the plane - though it would put the pilots in a bad spot similar to the existing MCAS issues.
Actually, one of the three test pilots did in fact lose the plane in the simulator. Hence the reclassification of the flight computer as a component with a catastrophic failure potential.
Look, I'm about the last person you're going to get to buy into the awesome safety of an aircraft that can't even pass a textbook test case for computing in adverse environments. If the bloody thing was so safe and high quality, it wouldn't have been able to suffer that induced failure in the first place. It would have been designed out. The fact Boeing was caught off guard by that failure means they weren't looking very hard, and were banking that no one else would either. Which is generally a really bad assumption to make, because Murphy finds a way. Every time.
Some quick examples - they could have sensors around gas quality but do not - they rely on the folks providing the avgas to have good fuel. They could do built in weight and balance calcs with sensors - they do not, they rely on poor estimates for that. etc.
Boeing in particular ALLOWS pilots to fly planes outside of most flight envelope limits. They do this in part because historically they didn't trust the computer as much and trusted pilots more.
Airbus by contrast doesn't let you do this unless you take air data totally offline or something crazy to force alternative law. I suspect boeing will be moving towards the airbus model which is probably right. Airbus had arguably less local myopia when designing their planes - ie, future pilot populations come from a big group and making the plane as error proof as possible is best. Airbus had some issues around their automation without pilot override initially but I think those are resolved by now.
If you want a scary video - check out this early A320 flying - first fly by wire / autothrottle:
https://www.youtube.com/watch?v=YAg-WauGrLU
Can't have been a good feeling as that pilot. Note that the A320 is now considered a very safe plane.
Aircraft have evolved from systems with almost no built in redundancy or safety. The design of the 737 is pretty old - 1967 or so. Let that sink in for a bit, consider what type of tech they had back in. This is not a 777 generation plane or even A320.
"unstoppable MCAS"
* Leave stab trim cut
* Hold electric trim up
* Control airspeed
* etc...
AoA senor is not a single point of failure. The 737 is not a fully automatic / flight protected plane. ie, the plane will continue flying with very little sensor input at all - you can even dispatch with problems with a surprising amount of equipment under the MEL.
You brought up faulty replacement parts yourself as a contributing cause, but when challenged you switch to methods to stop the unstoppable, undocumented MCAS. To which faulty part were you referring? The unimportant, non-SPOF AOA sensor?
In the US there were close calls, but fewer. If you beleive in the chain of safety better maintenance and pilot response may have saved the day a few times when faced with the MCAS problems. The safety reports from US pilots certainly sound scary (nose down on takeoff GPWS alerts etc).
I brought up maintenance as a contributing cause in the 737MAX crashes. I didn't realize I had been challenged on this. I would really encourage you to read some of the source reporting.
The FAA had a safety notice on this precise topic
"FAA has determined it is necessary to advise operators of the importance of performing proper operations and maintenance on AOA sensors”
This is the detail on the previous lionair flight before the crash flight - from wikipedia to save time:
"it had problems maintaining a constant altitude, with passengers stating that it was like "a roller-coaster ride".[109] The chief executive officer of Lion Air, Edward Sirait, said the aircraft had a "technical issue" on Sunday night, but this had been addressed in accordance with maintenance manuals issued by the manufacturer. Engineers had declared that the aircraft was ready for takeoff on the morning of the accident.[110][111] A later report claimed that a third pilot was on the flight to Jakarta and told the crew to cut power to the stabilizer trim motors which fixed the problem. This method is a standard memory item in the 737 checklist.[112] Subsequently, the National Transportation Safety Committee confirmed the presence of an off-duty Boeing 737 MAX 8 qualified pilot in the cockpit but did not confirm the role of the pilot in fixing the problem.[113]
Even worse is the history before that flight - REPEATED problems with the senor.
We are also now hearing that evidence submitted by lion air supposedly showing the proper fix for the sensor may have been faked.
https://www.seattletimes.com/business/questions-raised-about...
In terms of faulty replacement parts - my point was issues occur overseas that do not seem to happen within the US MROs. I'll do a quick recent example:
https://www.pressreader.com/south-africa/sunday-times-1107/2...
This follows the ex-president of south africa appointing a school teacher to head SAA (who also ran his private foundation and rumors around their relationship existed).
I'm not sure why you claim AoA is unimportant. In the chain of safety you want every advantage you can get, the more accurate data the better. So you want the AoA working. This reduces risks from MCAS. You are correct - AoA is not a single point of failure system (system is not working / plane falls from sky). There was a poster here claiming that it was.
Luckily if flying in the US the chance of the AoA being faulty in the first place would be a less I would argue than flying for lionair. My co-pilot might be quick to help. I might be quicker to get off automation and manually fly / trim.
Some of the self reports from the US specifically credit co-pilot assistance in averting a problem. See this one during takeoff - worst time to be heading towards ground at 1,500 FPM!
Within two to three seconds the aircraft pitched nose down bringing the VSI to approximately 1,200 to 1,500 FPM. I called "descending" just prior to the GPWS sounding "don't sink, don't sink."
Other side:
"PM's (co-pilot) callout on "descending" was particularly quick and welcome as I was just coming back to my display after looking away. System and procedures coupled with CRM (Resource Management) trapped and mitigated issue.
Synopsis
B737MAX Captain reported an autopilot anomaly in which led to an undesired brief nose down situation."
This was a full GPWS don't sink alert situation - very cleanly handled - the co-pilot BEAT the automated alert, the recovery was good. Should pilots have to deal with this? No. Did chain of safety work here? Yes - though closer to edge than one would like.
My own impression - US pilots much quicker to get off the automation / don't trust it as much / have more comfort or experience hand flying statistically maybe.
That was the assumption a year ago. Investigation has turned up tons of evidence against your null hypothesis. Crucially, this behavior was observed in the simulator and reported up the line.
> ... and a lack of backup sensors to offset the risk.
Not when multiple red flags were raised internally, and the backup sensors were sold as a premium upgrade.
> There will always be limits to how effective regulatory oversight can be.
Especially when we allow corporations to do their own regulatory "oversight"
Huh? The MAX was the most unsafe major new plane in ... something like decades. You can't make a statement excluding all the bad and then trying to say it's good. The bad is exactly what you're getting at here.
And even apart from that, the 737 is an obsolete, decades-old design. The only reason they're still making new iterations of it is because of type ratings requirements; if you actually wanted to make a great plane, you'd never base it on the 737. It has ancient analog gauge displays (which are now simulated digitally), doesn't use electronic flight controls, is too low to the ground leading to increased potential of foreign object debris into the engines, and as a consequence of this the newer more efficient jet engines don't even fit underneath the wings properly and hence had to be shifted forward into an unstable position that necessitated MCAS.
There's no two ways about it, there's nothing good about the 737 MAX, even apart from all the crashes.
As time went by and we learned more and the good idea faeries had time to think about things, we now have all kinds of regulation that makes the cost of making a new plane much higher if it is not an iteration on an existing design.
This sort of regulatory pressure has a tendency to nix innovation in favor of stability, and that's what we see here.
Aside from regulation, There are very good business reasons from both ends of the industry that exert pressure to iterate on an old design. (E.g Each additional airframe dramatically increases the complexity of an airline's maintenance organization.)
Boeing MBAs took over the company and drove quality down.
Maybe it's a side effect of financial capitalism taking over, maybe it's not but it's quite correlated.
[0] https://newrepublic.com/article/154944/boeing-737-max-invest...