How the Boeing 737 Max disaster looks to a software developer (2019) | Hacker News Reader

How the Boeing 737 Max disaster looks to a software developer (2019)

spectrum.ieee.org

141 pointsbehnamoh138 comments

The author makes the bizarre assertion that runaway trim is only caused by short circuits, and would only be continuous (not intermittent), and therefore the pilot would not recognize the MCAS runaway trim as runaway trim:

http://www.gregorytravis.com/WhyDoesNotPresent/

First off, as an engineer who often has to find and fix electrical short circuits, they'd be easy to find if they were continuous. They're frustratingly intermittent.

Secondly, I worked on the design of the 757 stab trim system. Runaway trim is the trim system coming on and trying to crash the airplane - it being intermittent is NOT part of the diagnosis. That intermittent vs continuous NEVER was part of the discussions about runaway trim. I don't know where that notion comes from. It doesn't make any sense.

Anyhow, the author certainly knows a lot about Cessnas. He stretches this into knowing about 737 systems, a much more complex airplane in its systems and aerodynamics, and he stretches it past the breaking point.

There is strong evidence against the assumption that this should be something that a typical flight crew can handle, in the form of the smoldering wreckage of not one but two planes, each with a trained, professional crew.

You're right that there's something wrong with the training. There were 3 flights that had MCAS failure. All three were able to restore normal trim with the electric trim switches, which override MCAS.

#1 turned off the trim system and continued the flight without incident. (This flight is almost never mentioned, as it doesn't fit the narrative.)

#2 never turned off the trim system and crashed.

At this point, an Emergency Airworthiness Directive was distributed to all MAX pilots instructing them to restore trim with the electric trim switches and then turn it off.

#3 turned off the trim system when it was severely nose down, and crashed.

>This flight is almost never mentioned, as it doesn't fit the narrative.

It was mentioned repeatedly Walter; if not in the articles, then in the comments. The fact remains that afterward, when regulators put more pilots (two military and one civilian trained) pilots into single-event upset simulations, 1 of the three failed to recover the plane in time.

If you acknowledge the training was faulty, then just let it be. Trim runaway presenting continuously vs. intermittently may not seem that important from a clean room engineer's perspective, but when viewed by a pilot through the lens of instruments, blaring alarms, and whatever else is going on in the cockpit, my engineer's optimism favors not relying on the human being perfect, or the defect only happening at a non-terminal part of the flight regime.

It sucks that it played out the way it did. Maybe if someone would have spent as much time on MCAS and the pilot training as you did the 757 auto-trim, it wouldn't have been an issue.

But they didn't. And innocents paid for it.

> 1 of the three failed to recover the plane in time

Without more information on what the background and training of those pilots, and why the one failed, I'm afraid it's just an anecdote.

In both crashes, the trim came on repeatedly and the crews repeatedly brought it back to normal with the electric trim switches. 25 times in the case of the LA crash. They obviously knew that the trim was the problem, because they took this corrective action. The trim running is obvious by the large black&white spinning wheels in the cockpit, and the deliberately loud clacking sound the running trim makes.

All they had to do was then turn it off.

> But they didn't. And innocents paid for it.

That's right, and that's the tragedy.

As to why they didn't, who knows. The stab trim cutoff switch is right there in a priority position on the console, because it's a damned important switch. Dealing with runaway trim is supposed to be a memory item, i.e. memorized.

It does suggest a training problem.

It suggests a system design philosophy problem. A system that continuously goes into a erroneous state instead of a safe state (turn off automatically and rely on manual trim) is designed badly. It couldn't be designed better, because cooperate added the requirement to remain within the flight model of the 737 at all cost.

Nobody is blaming the engineers for this. But it is, by design, a bad system, that can by attrition defeat human decisions.

Here's the article I was thinking of, though I think it was originally either in seattletimes or WSJ.

http://airsoc.com/articles/view/id/5d5143c7c4263ce96fd616b6/...

Differen source.

https://aviationweek.com/air-transport/software-fix-will-add...

The civillian trained test pilot lost it. The two military pilots did not.

Thanks for the reference. I haven't seen this information before.

"In one scenario, the bits chosen first told the computer that MCAS was engaged when it wasn’t. This had the effect of disabling the cut-off switches inside the pilot-control column, which normally stop any uncommanded movement of the horizontal tail if the pilot pulls in the opposite direction."

The cutoff switches cannot be disabled on the airplane.

"They flew the same fault scenario again, this time deliberately allowing the fault to run for some time before responding. This time, one of the three pilots didn’t manage to recover and lost the aircraft."

That's kinda vague.

https://www.seattletimes.com/business/boeing-aerospace/newly...

There it is! Knew it was in SeattleTimes somewhere!

And lookout for the part about disabling cutoff switches, I think Dominic might have written that better, because I think he's talking about a safety function that was removed specifically for the MAX where hauling back hard on the yoke would stop trim commands in progress on NG airframes? MCAS required that that functionality not be ported forward to the MAX.

This isn't disabling those cutouts though. Those are literally, like you said, a hard switch.

>According to a second FAA source, it was the AEG pilot, representing a typical U.S. airline captain, who failed to recover the jet.

There's the bit about the civilian test pilot.

Thanks. My experience is Aviation Week is a much more reliable source on technical topics.

It's a hard switch for good reason. I hate "soft" switches that supposedly disable microphones. Phooey on that.

BTW, on my hotrod I installed a hard switch to shut off the fuel pump, as well as an oil pressure switch that would shut it off. I have a g-switch to cut it off, too, but haven't gotten around to installing it.

The article also does not mention the LA flight that survived the MCAS malfunction. Hence it always fall to me to mention it.

Flight into wind shear does not always lead to crashes. This does not mean that there is no need to train pilots for it.

If the trim runs away far enough (which it clearly can, more easily because MCAS was also activating the stick shaker which is INCREDIBLY DISTRACTING) it can become impossible for a pilot to physically turn the manual trim wheels to get back into trim.

So the procedure for runaway trim (turn off electric trim, use hand wheels) doesn't work. You have to instead use electric trim first to get back into normal trim, then turn off the electric trim, and never turn it back on for that flight.

> ... it can become impossible for a pilot to physically turn the manual trim wheels to get back into trim

I feel like this isn't talked about enough. I had no idea how bad it can actually get: https://youtu.be/aoNOVlxJmow?t=806

The difficulty of using the manual trim wheels is also demonstrated in this simulator run showing the MCAS problem: https://www.youtube.com/watch?v=xixM_cwSLcQ&t=1011s

There is some discussion amongst pilots at https://www.pprune.org/tech-log/619326-boeing-advice-aerodyn... which further explains how difficult it can be to manually reduce load on the stabilizer and manually trim the nose up.

I did some napkin math to try to calculate the force that'd have to be applied to the jackscrew to actuate under load, and it was not a small number. Basically, an electric motor with the torque of a sporty automobile.

With the gear ratios we're talking with by hand actuation, I would not want to have to be one of those pilots.

> It does suggest a training problem.

Aviation rules are written in blood. Training is being updated and fixed. But regardless, MCAS itself is controversial and fallible - it’s crucially important that Boeing make it better.

IIRC, #1 also had an extra pilot at hand, and that extra pilot is the one who suggested turning off the trim system.

> #1 turned off the trim system and continued the flight without incident. (This flight is almost never mentioned, as it doesn't fit the narrative.)

I.e. they piloted the plane without the MCAS, which they were not trained to do. The whole reason of the existence of the MCAS is to avoid training the pilots to fly the planes without it.

>I.e. they piloted the plane without the MCAS, which they were not trained to do. The whole reason of the existence of the MCAS is to avoid training the pilots to fly the planes without it.

This kind of hand wringing shares a non-negligible part of the blame for why hundreds of people are dead.

If Boeing had been allowed to say "of course our new engines are gonna pull the nose up a little harder, throttle back and climb slower if you don't like it" the MCAS system would never have been built to be the safety critical boondoggle it became. The plane itself handled just fine but the problem that they tried to solve with MCAS is that it wasn't identical to the prior 737. Even the most incompetent professional 737 pilot can fly a 737 that wants to <insert control input here> slightly when <insert other control input here> so long as they're expecting it (remember, they train for this stuff in simulator and most of these pilots are certified for multiple different aircraft) but just letting the pilots deal with it wasn't an option because the regulations disallowed that due in large part to decades of "people are just dumb cogs" type thinking.

And once the airplanes started falling out of the sky nobody could say "just turn of MCAS, it won't handle the same but close enough" because Boeing, the airlines and the FAA were all so balls deep in the lie they'd look like idiots if they changed their opinions and thanks to the society we've cultivated nobody is willing to be the first to break formation because that's who gets targeted by the bulk of the fallout.

That is not correct, it simply would not be certified as airworthy without it. Differences in feeling between airplanes are fine. Airbus has a system of common type ratings allowing fairly quick transitions. Especially between A330, A340 and A350.[0]

> MCAS is a longitudinal stability enhancement. It is not for stall prevention (although indirectly it helps) or to make the MAX handle like the NG (although it does); it was introduced to counteract the non-linear lift generated by the LEAP-1B engine nacelles at high AoA and give a steady increase in stick force as the stall is approached as required by regulation. > The LEAP engine nacelles are larger and had to be mounted slightly higher and further forward from the previous NG CFM56-7 engines to give the necessary ground clearance. This new location and larger size of nacelle cause the vortex flow off the nacelle body to produce lift at high AoA. As the nacelle is ahead of the C of G, this lift causes a slight pitch-up effect (ie a reducing stick force) which could lead the pilot to inadvertently pull the yoke further aft than intended bringing the aircraft closer towards the stall. This abnormal nose-up pitching is not allowable under 14CFR §25.203(a) "Stall characteristics". Several aerodynamic solutions were introduced such as revising the leading edge stall strip and modifying the leading edge vortilons but they were insufficient to pass regulation. MCAS was therefore introduced to give an automatic nose down stabilizer input during elevated AoA when flaps are up.

http://www.b737.org.uk/mcas.htm#background

[0]: https://www.airbus.com/en/products-services/commercial-aircr...

This is the part that bugs me as well. When you have to disable MCAS, you're flying a plane you've not trained for. Particularly given the fact MCAS featured prominently in the landing regime.

"The whole reason of the existence of the MCAS is to avoid training the pilots to fly the planes without it."

That summarizes why the plane did not go through a new type certification process - avoid costs of pilot training. Boeing instead painted over the differences (larger fans mounted forward) with software to make it feel the same.

The classical problem in software and apparently in engineering is that some suit with a big mouth makes promises that can't be kept, and then makes everything difficult for everyone else by trying to make their lie true. Either people sacrifice their health, sanity and families bending over backward trying to cover for them, or some safety problem makes it into the product due to false constraints.

The 737 Max was sold on not having to do certain kinds of retraining. "Something wrong with the training" is a divide by zero error here. There was no training. If memory serves this was also tied up in how much and what kind of recertification the FAA would require for a 'new' plane versus a refit of an existing one. Training means newness, and newness means retesting. The lies snowballed into dead passengers.

I don't think GP is suggesting there was some issue with the training. I think they are suggesting that it's very difficult to justify MCAS (as originally implemented) as WAI when it quite clearly brought down two airplanes full of people.

This seems a bit of a non sequitur from your first comment, but you did reply (here). I'm still not sure what your point is, though. What "narrative" are you referring to?

>The author makes the bizarre assertion that runaway trim is only caused by short circuits

That's not the only problem with TFA. I worked on the Hercules, not any of the Boeing airliners, but this "30-year pilot" has some pretty fundamental misunderstandings of how MCAS works, what the effects of moving the engines are, and what actually went wrong with the 737 Max. Color me surprised.

Regurgitating the same "MCAS prevents the aircraft from stalling because it's unstable" shlock betrays the author's surface level (at best) understanding of the issue.

As a surface-level-understanding layman myself, what about this is wrong? The 737 MAX is unstable, no?

>The 737 MAX is unstable, no?

No, it's not unstable. The 737 Max exhibits stable pitch behavior along the entire envelope. A lot of people think "unstable" means "it crashes" but these terms have precise meanings.

The 737 Max is a stable airplane. One of the problems it exhibits is that with increasing angle of attack, the stick force does not increase (linearly is the requirement, I think, I'm writing from memory here).

The way you know this is because MCAS moves the horizontal stabilizer directly, and not the control column that the pilot is holding. An anti-stall device like a stick pusher or a stick shaker provides tactile feedback to the pilot via the control column. Moving the horizontal stab directly is (basically) altering the trim of the aircraft. Thus it is a handling-qualities system.

I see. So calling it "unstable" isn't really fair. It's more that it just flies differently enough that it would feel unfamiliar to typical 737-trained pilots.

Correct, but it's more subtle than that. FAR's are prescriptive. To carry passengers a plane from the pilot's perspective must have that linear increase in stick pressure required to pull into a stall, otherwise it can't be certified for carrying passengers.

This is part of why the MCAS could not be decoupled from the flight computer as I recall. If you turned off MCAS, you'd be non-compliant in the wind-up turn flight regime, and when coming in on final if I recall correctly.

So in aerodynamic parlance it is stable, but with disqualifying flying characteristics that had to be mitigated via a gadget whose nature, failure modes, and existence was largely hidden from the pilot, and downplayed in the FMEA/FMECA. (Failure Mode, Effects, and Criticality Analysis)

This downplaying was specifically intended to keep regulators from stipulating enhanced training for pilots. Thereby saving money.

This is a good summary.

Not getting pilots retrained was the cost saving strategy for companies buying the new area aircrafts.

Not needing retraining was a good goal because it improves safety. Sometimes airplanes have crashed because in a crisis the pilot did the right thing for another airplane he was trained on, but was the wrong thing for what he was piloting at the moment.

It's not just a cost saving measure.

MCAS was a sound concept. It's implementation, however, was not sound at all.

A typical pilot switches aircraft type multiple times in their career. It's all about the cost and convenience ultimately, allowing same pilots to fly both 737MAX and the older 737NG as most 737MAX customers operate both.

As far as MCAS goes, in my opinion Airbus got it right already in the 80's in their A320 with their flight envelope protection.

Do you understand how delusional it sounds in this specific context to say that not needing retraining improves safety? Not that it might improve safety or should improve safety, but that it does improve safety?

"Not needing retraining" is another way of saying "works as expected".

Yes, it would likely have been safer if the 737 Max worked as expected by the pilots flying it.

Rightly said!

> Not needing retraining was a good goal because it improves safety.

That only works if retraining is not actually needed. Boeing's own test program gave adequate (but undisclosed, at the time) information for an independent observer to conclude that this was not the case here.

I'm not sure I could agree that MCAS is a sound concept. Moving the trim automatically to account for the pitch up at high angles of attack feels like a hack, like restarting a dodgy service every couple of hours rather than fixing the underlying issues.

I've never understood why the fix wasn't implemented in the elevator feel computer. Except perhaps that it doesn't have AoA inputs and it's an old hydroelectrical system for which there may be a lack detailed knowledge.

No, it's always stable. The issue is that pulling back further on the control stick wasn't causing an increase in force feedback at high angles of attack, making it more likely for the pilot to inadvertently stall the aircraft.

> That intermittent vs continuous NEVER was part of the discussions about runaway trim. I don't know where that notion comes from. It doesn't make any sense.

It comes from Boeing's checklists: https://i.stack.imgur.com/cRG4J.jpg

It makes perfect sense, because continuous operation identifies a fault condition. Under normal conditions, automatic systems may intermittently move the stabilizer, but never continuously. It may be faulty when moving on its own, but if it moves continuously without input, it's definitely faulty. That's where the name cames from: runs away into physical stops.

Weird part is that IEEE published this. Someone might actually take it seriously...

> Apparently the 737 Max pitched up a bit too much for comfort on power application as well as at already-high angles of attack. It violated that most ancient of aviation canons and probably violated the certification criteria of the U.S. Federal Aviation Administration. But instead of going back to the drawing board and getting the airframe hardware right (more on that below), Boeing relied on something called the “Maneuvering Characteristics Augmentation System," or MCAS.

Nope, this is 100% false. The purpose of MCAS is to give the MAX the same aircraft handling characteristics so that it retains the original 737's type rating. Thus 737 type rated pilots can fly the MAX under their 737 type rating. A big part of selling the MAX is type rating commonality with the airlines existing 737-train pilots. Juan Browne, a real 777 airline pilot, mentions this in one of his excellent YouTube videos (https://www.youtube.com/user/blancolirio).

> I have been a pilot for 30 years

I suspect this person is a private pilot with no actual airline aviation experience.

Edit: This is the same guy that boosted his aviation credentials by citing that he's flown a 757 simulator (https://www.eetimes.com/software-wont-fix-boeings-faulty-air...). So yes, he's a GA pilot that has flow in some aircraft simulators.

> Among Boeing’s critics is Gregory Travis, a veteran software engineer and experienced, instrument-rated pilot who has flown aircraft simulators as large as the Boeing 757.

> The purpose of MCAS is to give the MAX the same aircraft handling characteristics so that it retains the original 737's type rating.

The article mentions that prominently in a few paragraphs, this is the first one:

“That's because the major selling point of the 737 Max is that it is just a 737, and any pilot who has flown other 737s can fly a 737 Max without expensive training, without recertification, without another type of rating.”

The article mentioned something that I didn't know about before: That the yoke feel is computer-generated and that MCAS was messing with that.

It also correctly points out that "with the 737 Max, Boeing has changed philosophies about human/machine interaction" - they didn't just hide a system in the plane that changed the planes behavior without telling the pilot, they fundamentally changed the basic assumptions pilots make, and still called it the same plane that doesn't require separate training. I'd consider that an even bigger crime than the other recklessness that ultimately led to the crashes.

I think that part about the yoke feels is incorrect. MCAS only changes the trimming and that changes the yoke force required when trying to maintain the same angle of attack.

This article highlights a few fundamental flaws:

- design certification treated as "just another 737" which ignored fundamental deviations in hardware and software, complicated by safety certification by manufacturer instead of public employee

- hardware design including a "dynamic instability" in which airplane approaching an aerodynamic stall had a tendency to go further into the stall due to lift produced by the oversized engines at high angles of attack, which was intended to be mitigated with software

- omission of using multiple inputs, including the opposite angle-of-attack sensor, in the computer's determination of an impending stall

- a changed philosophy about human/machine interaction from humans winning a battle of the wills every time to computers winning a battle of wills in cases of envelope protection

The final item is perhaps the one most fundamental to other cases of safety critical human machine interaction designs. If we are signing over agency to machines for envelope protection, that means we need in advance to understand every potential edge case scenario where that envelope may be mis-framed. Such comprehensive foresight in some environments may be intractable. For the 737 this was exasperated by the presence of an inherent source of instability originating from hardware design.

> Those people have decades of institutional memory about what has worked in the past and what has not worked. Software people do not.

> The people who wrote the code for the original MCAS system were obviously terribly far out of their league and did not know it.

Hmmm... having met a few embedded aerospace software engineers, I'm not sure I would be pointing the finger in their direction, although I would love an insider account. All public information seems to indicate these were decisions made at the executive level.

I'm sorry, but no. Such criminal negligence isn't excusable by "decisions were made higher up". Human lives are at stake with aviation, the engineers implementing this should have known better and refused.

I think you're really discounting the poisonous influence that management can have here. As a Quality Assurance person, my number one mental tic is the "I told you so" that inevitably follows an executive override.

I'm not discounting manager influence, I'm just saying that for such a matter the engineers should have refused.

Maybe they fear losing their jobs. As I understand it, with the healthcare system, this can itself be perilous.

Indeed, i had forgotten that nice part of the American labour system.

They could have blown the whistle anonymously then.

Sounds like a nice way to end up in a plea bargain.

Don't forget with the consolidation in that space, there isn't exactly a large swathe of competitors to get hired by.

It is bizarre to expect an embedded software engineer, who is coding from a specifications document, to know enough about aircraft control, control dynamics, pilot training, aerodynamics, etc to somehow figure out MCAS was a somewhat poorly designed/communicated "workaround".

It was the embedded software engineer (s) who didn't do any redundancy. Knowing this is for an airliner, how on earth do you reconcile the two?

Safety really is a systems level concern. Aircraft software should implement the requirements the systems level put on it and nothing else. If the specification is somehow insufficient this is fed back to the systems level for evaluation and requirements have to be added/changed. This is the entire point of DO-178C.

> Everything about the design and manufacture of the Max was done to preserve the myth that 'it's just a 737.'

Do we know what is next for Boeing in that same segment? Are they planning to come up with another 737 variant after the MAX, or starting from a completely different platform[1]? Did they disclose anything?

[1](I mean from a technical standpoint - not marketing/naming)

The 737 Max is already abusing the platform beyond what it could handle, and it's still less efficient than the A320neos, and even less so than the A220s. Boeing need a new short-haul platform, like a decade ago :)

Past discussion has been prolific... 3 years ago, 528pts, 297 comments[0], a year ago, 51pts, 72 comments[1]

[0]: https://news.ycombinator.com/item?id=19694207

[1]: https://news.ycombinator.com/item?id=24233339

> For the life of me, I do not know why those two basic aviation design considerations, bedrocks of a mind-set that has served the industry so well until now, were not part of the original MCAS design.

Because that would force retraining? “Sometimes the MCAS system will disagree, if that happens it will disengage and you are no longer flying a 737, good luck!”

> Pitch changes with increasing angle of attack, however, are quite another thing. An airplane approaching an aerodynamic stall cannot, under any circumstances, have a tendency to go further into the stall. This is called “dynamic instability," and the only airplanes that exhibit that characteristic—fighter jets—are also fitted with ejection seats.

I still don't get how this design was ever allowed to happen or continue to happen. Passenger jetliners shouldn't be dynamically unstable.

The 737 Max is not aerodynamically unstable to the point of causing these crashes. Any airplane can stall in any attitude or at any airspeed. The Max is not inherently unstable or aerodynamically unsuitable for flight.

The Max could fly without MCAS just fine, but pilots would have to be re-trained. Airlines didn't want to foot the bill for such an expense so Boeing applied the Maneuvering Characteristics Augmentation System (MCAS) to literally augment the maneuvering characteristics of the Max to match the 737-NG, which pilots are already familiar with.

Bugs in the system then led to runaway trim which caused a stall and crashes.

Well that's not true either. Yes it's true that it is aerodynamically stable, and could fly without MCAS just fine.

But it isn't just an issue of retraining. To prevent inadvertent stalls, FAA regulations require increasing control stick force feedback - a requirement which the Boeing 737 Max did not meet without MCAS.

I absolutely accept this, in a world were deep technical explanations aren't exactly in fashion, it's the only explanation which fits the available evidence. However, it also runs contrary to the official / media friendly version of what Boeing say happened.

I wish we had a good source, the best I've found is the 737 technical site, I wish it was available in an official report.

It actually was if I recall.

https://www.faa.gov/foia/electronic_reading_room/boeing_read...

Page 10:

>The 737 MAX was designed to handle and feel the same to the pilot as the 737 NG. Without the MCAS function, in some small areas of the flight envelope — such as approaching a stall and during higher g-force maneuvering — the new engines contribute to the control column feeling lighter in the 737 MAX than the regulations allow. These are not areas of the flight envelope in which the airplane normally operates. However, FAA regulations - specifically 14 CFR 25.143, 25.201, 25.203, 25.251, and 25.255 - still require the control column to have a higher pull-force feel in these flight regimes than would exist on the 737 MAX without the added stability from the STS and MCAS function.

Good find! Interestingly I remember that being a relatively well accepted explanation here before the Ethiopian crash which is before that was published. Though I'm sure similar wording appeared elsewhere.

> The LEAP engine nacelles are larger and had to be mounted slightly higher and further forward from the previous NG CFM56-7 engines to give the necessary ground clearance. This new location and larger size of nacelle cause the vortex flow off the nacelle body to produce lift at high AoA. As the nacelle is ahead of the C of G, this lift causes a slight pitch-up effect (ie a reducing stick force) which could lead the pilot to inadvertently pull the yoke further aft than intended bringing the aircraft closer towards the stall. This abnormal nose-up pitching is not allowable under 14CFR §25.203(a) "Stall characteristics". Several aerodynamic solutions were introduced such as revising the leading edge stall strip and modifying the leading edge vortilons but they were insufficient to pass regulation. MCAS was therefore introduced to give an automatic nose down stabilizer input during elevated AoA when flaps are up.

http://www.b737.org.uk/mcas.htm#background

The following debacle of only relying on one input, trying (and still trying) to blame the pilots, a 1960's designed system, not declaring it as a safety critical piece to reduce review and so on is well documented.

> Bugs in the system

And most of all, and the truly inexcusable part, zero redundancy.

Then the correct and simple solution is to rip out the MCAS and re-train all the pilots.

As several comments here have stated, it's unlikely that the 737 Max without MCAS would be a certifiable airplane. In a way MCAS makes the Max fly more like the NG but only in the sense that it makes the Max fly more like any other certifiable airframe.

There are plenty of aircraft with common type ratings which are more different than the Max and the NG. As an example the 757 and the 767 share a common type rating.

Okay so then the stated objection that I quoted is technically incorrectly, but the spirit is the same. This is not a well-engineered plane and it requires the MCAS to fly it, and without the electronic band-aid the flight envelope would never have been acceptable.

So it might not exactly be a fighter jet but its outside of the bounds of passenger airlines.

this just isn't true. Airbus has used tricks like this for decades without problems. the problem was just a totally incompetent implementation.

The runaway trim was driving the nose down into a dive, not up into a stall.

Ah, right. MCAS thought there was an impending stall and took action to prevent it.

Correct, because the AoA sensor was indicating the plane was busy belly flopping into the airstream according to one flight or another's Flight Data Recorder.

All planes are only stable within certain flight parameters. The problem wasn't that the plane was unstable, it was that the stability region was different than that of previous 737s. If the plane had been given a different name and required simulator training, none of this mess would have ever happened.

(Or if they'd just correctly classified MCAS as a safety critical feature and made it have redundant inputs).

Or if they’d just done a sanity check on the AOA data.

You're right, but that's a different issue from requiring the airplane to be dynamically stable in all its flight modes.

> Passenger jetliners shouldn't be dynamically unstable.

Well, they are. The active yaw damper is required by the FAA.

https://en.wikipedia.org/wiki/Yaw_damper

Where does that wiki say that the FAA requires the active yaw damper?

"several airliners were deemed to be unsafe to fly without an active yaw damper."

The string "FAA" does not appear in that wiki nor the cited article. The cited article just says that descent to FL260 is required in one model of 727 if one of the yaw dampers fails.

The cited article doesn't even seem to support the line in the wiki. I don't see the word "unsafe" anywhere in the cited source and the only "requirement" is the descent to FL260 in case of a failure.

Anything deemed unsafe has the FAA involved in it.

You might also be interested in fatal crashes resulting from yaw damper failures:

https://en.wikipedia.org/wiki/Dutch_roll#Accidents

Amusingly enough, the 737 is one of those few airframes which are dynamically stable in yaw. The yaw damper is not on the required equipment list. Others like the 757, the failure of one of the two fitted yaw dampers is enough to prevent dispatch.

What an insightful article! Reading through this gave me a feeling of dread as I imagined the pilots pulling as hard as they could on the yoke (?) and being overpowered by the computer.

That is incredibly tragic.

On both the LA flights, and the EA flight, the crews restored normal trim by simply using the electric trim switches, multiple times. One of the LA flights recognized runaway trim, and turned the trim system off after restoring normal trim, and landed without further incident. The other two flights did not.

The EA flight received an Emergency Airworthiness Directive instructing them to (1) restore normal trim with the electric trim switches and then (2) turning off the trim system with the cutoff switch. They did not follow those directions.

Runaway stabilizer trim is the trim motors coming on and driving the stabilizer in an adverse direction. This is exactly how the MCAS failure presented itself. The author tries to claim it isn't really runaway trim, an argument which doesn't make sense.

Are you trying to say the MAX was safe to fly with that EAD in place? If so, why did Boeing go to the trouble of redesigning MCAS?

It looks like you are pushing a certain agenda.

Anyway, just because the pilots "could have" recovered the plane(s) they were flying, does not make the design of MCAS sound. By all accounts the MCAS design was botched (single source of inputs, no sanity check on the inputs, etc.). This does not seem to be a point of controversy. Plus, as others will point out, two hull losses are a grim testament to this.

The agenda I'm pushing is the facts about the crashes that are consistently omitted from popular narratives about it.

Other times this has come up, I faulted Boeing for making several design errors in the MCAS.

But the pilots bear some responsibility, too, because they did not follow runaway trim emergency procedures. In order to make flying safe, all causes of an accident (and there are usually many for each accident) must be addressed.

You must excuse me but I need to be blunt here;

Your agenda is throwing "shade" at Pilots and insinuating that their responsibility is much larger than it actually is.

I am not sure why you keep pushing this given that you yourself are not a Pilot. Yes, you have listed your credentials but they are not a substitute for the actual job of "Piloting".

As a layperson starting with the collection of data from https://en.wikipedia.org/wiki/Boeing_737_MAX_groundings (which also contains a link to this article) it is very clear that Boeing is fully to blame for NOT disclosing the changes done to the 737 MAX system and how different it was from what the Pilots were used to. The Pilots were shown and assured a facade (i.e. everything works the same as what you know and trained on!) but when things went wrong they had no training to handle the new components behind the facade (they were not even told about these!).

It is inexcusable and Boeing bears full liability.

> The agenda I'm pushing is the facts about the crashes that are consistently omitted from popular narratives about it.

So you're saying that out of 3 flights with this issue occurring: 2 have crashed and 1 has been recovered by the crew. And this supposed to be a supporting point? So 66% failure ratio due to whatever circumstances?

Everywhere in this thread you are saying that MCAS was a good concept, the execution wasn't. As a software person, albeit not in aviation but worked with piston aviation engines engineers (so I kinda understand the whole concept of airworthiness and what goes into the certification process), I have to say that there are many ideas one comes across as sound on paper but should never be done just because they can be done.

The other thing which really raises my eyebrows is the claim, and I'm sorry if I'm misinterpreting what you are saying, that the EA and LA crews were able to apply the correct procedure multiple times yet the aircraft still crashed?

> But the pilots bear some responsibility, too, because they did not follow runaway trim emergency procedures.

So here's a question: would that single procedure justify the need to retrain pilots? Why would pilots fail to execute this emergency procedure if the aircraft was flying like the original 737 or NG? Would the crew be required to execute the same procedure on earlier 737's in a similar scenario?

Wow!

That's the best read of what's going on with the 737 Max system and the MCAS software that helps to drive it.

Totally understandable now.

The author is a Cessna pilot. It's very, very different, from a jet airliner. High altitude flight with swept wings has little in common with a low-and-slow straight wing Cessna.

http://www.gregorytravis.com/About/

He also writes about runaway trim: http://www.gregorytravis.com/WhyDoesNotPresent/ and assumes that Cessna training on stab trim is the same as for a jet airliner.

It isn't.

He also seems quite unaware that the flight previous to the LA flight that crashed also had MCAS activation, recognized it as runaway trim, and recovered by simply turning off the stab trim.

P.S. My dad was in the Air Force during the transition from propellers to jets. He said there were a lot of crashes from expert prop pilots thinking a jet was just a faster and higher flying propeller machine.

Jet pilots require substantially different flight training.

> The principle of Carnot efficiency dictates that the larger and hotter you can make any heat engine, the more efficient it becomes.

While this is true, for air-breathing propulsion engines there's another factor: It is more efficient to accelerate a lot of air a bit, than a bit of air a lot. That's why the jet engine was replaced by the turbo fan with increasingly higher bypass ratio: they have a jet engine in the middle, but the fan also pushes through a lot of additional air that passes around the turbine itself entirely.

In particular, the propulsive efficiency under some simplifying conditions is 2/(1+v_ex/v_in): the higher the exhaust velocity the worse the efficiency.

https://en.wikipedia.org/wiki/Turbofan

https://en.wikipedia.org/wiki/Propulsive_efficiency#Jet_engi...

I wrote about how my experience in airliner mechanical design influenced the D programming language:

Origins of the D Programming Language https://dl.acm.org/doi/pdf/10.1145/3386323

Software development can learn a lot from aviation designs.

Link appears to be broken for me. D is next on my GCC safari of programming languages to muck with.

I just tried it and it works. I have no idea why it doesn't for you.

Now it works...

I will not question this.

That reminds me of openssl, which tends to release 1.0.2a, 1.0.2b, 1.0.1u and pretend it's the same version to not have to go through another audit.

They fixed that and moved to semver, and the FAA should put limits on what can be "technically the same version" too.

Wow, what a fantastic article. Many thanks to Gregory Travis for writing this.

It seems we're learning the same lessons over and over again about the nature of failure in complex systems. It seems reasonable to me that we'll observe more of these failures as our world becomes increasingly complicated, particularly in social and economic systems.

Perhaps we should be teaching the fundamentals of complex systems in high-school just as we teach basic science? It seems to me there is a distinct lack of appreciation of the nature of complex systems in our culture at a general level.

Just as some people think it would be better to learn statistics than calculus in high school - I wonder if it would be equally important to learn about complex systems?

I'm interested to hear what others think.

There is a bunch of FUD being thrown around in this thread and in the interest of Truth, here is a PSA;

1. Use the Wikipedia Page https://en.wikipedia.org/wiki/Boeing_737_MAX_groundings as your starting point for understanding the details. This article and more and better ones are all listed here.

2. This article from The Verge is the best for the layperson to get the overall picture of all the issues involved: https://www.theverge.com/2019/5/2/18518176/boeing-737-max-cr... Note that "Human Error" mentioned here refers to everything from Boeing Management, Marketing, Sales and Training.

3. Article on Department of Transportation Inspector General's scathing report on how Boeing misled FAA and everybody else : https://www.flightglobal.com/airframers/inspector-general-sl...

4. Remarks from "Allied Pilots Association" : https://www.businessinsider.com/boeing-737-max-apa-head-ques...

Boeing's previous suggestion that pilot error may have caused the crashes "offended" members of the APA, Carey said, as he criticized suggestions from some in the media that training standards may have been lower in the countries where the crashes occurred.

"To make the claim that these accidents would not happen to US-trained pilots is presumptuous and not supported by fact. Vilifying non-US pilots is disrespectful and not solution-based, nor is it in line with a sorely needed global safety culture that delivers one standard of safety and training."

He also said that Ethiopian Airlines has a simulator for the 737 Max, while no US airlines do.

5. Blaming Dead Pilots by Boeing and its lobby in Washington : https://www.corporatecrimereporter.com/news/200/blaming-dead...

The background of this is covered in the new book, Flying Blind:

https://www.amazon.com/Flying-Blind-Tragedy-Fall-Boeing/dp/0...

It talks about GE a lot as well, as both companies were destroyed by the Jack Welch "school of managers".

I'm very surprised this article remains published. It directly contradicts the actual narrative of events (including the EAD that, if followed as by the first flights with recorded MCAS issues, offered a prevention for the subsequent crashes) and unfortunately perpetuates incorrect definitions and understandings of runaway trim and unstable airframes.

> including the EAD that, if followed as by the first flights with recorded MCAS issues, offered a prevention for the subsequent crashes

The one that was followed by the Ethiopian pilots and the plane still crashed, resulting in the grounding of the 737 Max by all aviation authorities bar the FAA?

It wasn't followed by the Ethiopian pilots.

For reference:

Boeing Emergency Airworthiness Directive

"Initially, higher control forces may be needed to overcome any stabilizer nose down trim already applied. Electric stabilizer trim can be used to neutralize control column pitch forces before moving the STAB TRIM CUTOUT switches to CUTOUT. Manual stabilizer trim can be used before and after the STAB TRIM CUTOUT switches are moved to CUTOUT."

https://theaircurrent.com/wp-content/uploads/2018/11/B737-MA...

I highlighted the omitted action.

You keep saying this every time the 737 Max comes up on HN. I don't know why you still can't understand that "can be" has a very different meaning from "must be", and the Ethiopian Airlines pilots followed the procedures as written. The Boeing airworthiness directive was incredibly poorly written, and Boeing is to blame for this, not the pilots who attempted to follow the directive as written.

It also didn't mention that you had 5 seconds after the release of the electronic trim switch to flip the CUTOUT before MCAS poured back on the trim, but as I recall that took more digging to find out afterward.

This was also not pointed out in any type of pilot accessible documentation either; nor did it apparently make it's way to the FAA; as I recall them mentioning it was supposed to at max apply .6 degrees trim at once, but was later changed to ramp up to 2.5 degrees trim repeatedly, without an updated safety analysis.

Paperwork update failure:

https://www.avweb.com/recent-updates/business-military/mcas-...

Electric stab trim switch reset of MCAS activation timer was mentioned in the final report.

https://www.faa.gov/foia/electronic_reading_room/boeing_read...

Both crews repeatedly used the trim switches to bring the aircraft back to normal trim. The cutoff switch was within easy reach, easily set within 5 seconds. And even if it took them 6 seconds, or 7, the trim moving does not mean it's instantly hard over and unflyable.

You can also turn off the trim system while using the trim switches giving a 0 second delay. The reason is the circuit turning on the trim is a wire with the trim switch and the cutoff switch in series. Both have to be on for current to flow. Think of it like a light switch in a room and the corresponding breaker in the breaker box. Both have to be "on" for the light to come on.

The evidence that this is quite possible is the first LA crew experiencing MCAS malfunction trimmed back to normal, turned it off, and landed safely.

Oh, absolutely. I don't doubt that. Though there was a subtle change to the meaning of the switches.

https://www.seattletimes.com/business/boeing-aerospace/boein...

In their eagerness to match the procedure, they actually lost a level of granularity in system isolation. The infamous "electrical short" misunderstanding, I think, may have risen from system description that supported the change. The idea being a short in the yoke switches could cause a runaway, which is what the left switch in the NG is for isolating, and the right switch was for isolating the Autopilot.

In MAX, this distinction was lost. Either switch completely isolated both Autopilot and yoke switches.

Assumedly, if one was very studious and aware of the switches purpose in the NG, one could get taken by surprise flying the MAX if they expected them to work the same way.

This is why I tend to beat into my devs the importance of "if you change the interface, you explain the change, or you offer the ability to fall back to older behavior.

I realize that's not practical in aerospace's regulatory environment, but it is nevertheless a helpful guide.

I bring it up because nobody else ever does.

If they did not understand the EAD, all of it, it was their job to get clarification.

P.S. At least you read it. That's nice to see, and rare.

Like all the pilots, including from Lion Air, that requested extra training from Boeing and were laughed at?

Asking for clarification is not like asking for more training.

But after Boeing laughed at their "incompetence" for asking for training, how could they confidently ask for clarification?

Someone that unconfident shouldn't be a pilot.

Besides, consider that overhead is a matrix of circuit breakers. Their purpose is so the pilot can individually shut down each system. They work just like the breakers in your house.

The stab trim, however, is so important the switch is right there on the console within quick & easy reach. It's still a breaker, though. It works by cutting off all power to the trim motors.

It's really not rocket science. It's a freakin' switch.

If you click on past you'll see the previous huge discussions on this article

It looks an awful lot like what I expect to happen when critical code is outsourced to lowest-rate offshore contractors like HCL.

(2019)

This was a terrific article, I remember reading it maybe two years ago. As a self-taught software developer, I've noticed a critical difference between those with a degree fresh out of school and those with experience, a difference which is touched upon in the article. Those with experience know when to say "to implement that, I need to study". But those with a fresh degree rarely know to say "to implement that, I need experience". The author directly accuses software developers and software management inexperienced in the field of aviation as a cause of the failures.

The author's name is Gregory Travis, and this abridged paragraph stands out to me:

  > Another difference is between the autopilots in my system and that in the 737 Max...
  > ...the system simply goes off-line and alerts the pilot that she is now flying manually.

Why would the author refer the themself in the female? Is Gregory a female name too, or is this some literary device that I don't understand?

>Why would the author refer the themself in the female?

It's a thing that has become more popular to get away from assumptions that one sex or another is predominantly associated with some manner of activity. I've noticed it in several different authors writings. Or it could be that that by 'she', the author is referring to their aircraft, which I believe traditionally is considered a "she" just as a ship or boat is.

Yes, I agree in principal any hypothetical character in a narrative should be just as likely to be "she" as "he", but in this case the author specifically mentions his own personal aircraft. Maybe this was an attempt at such (justified) political correctness but being sloppy about where it was applied.

Reply on news.ycombinator.com