Behind the Lion Air Crash, a Trail of Decisions Kept Pilots in the Dark
nytimes.com
nytimes.com
So the interface didn't change, and the procedure's the same. Should Boeing and airlines update training every time they change something "under the hood", even when the procedure for pilots is the same? How about when they make software updates to already-flying models?
This seems to be exactly the interface change that lead to the crash.
The real question here is how commonly this feature was used. If it was common, then not putting it in the manual is a big problem!
You can always override cruise control by stomping hard on the brake (like to avoid an imminent crash). That's how it's always worked and you've gotten used to this, and done it on occasion when warranted.
Now imagine that the next generation of adaptive cruise-control/"auto-pilot"/whatever comes out, and stomping on the brakes no longer does anything. You have to first disable the cruise control by pressing a button on the steering wheel, and only then will inputs to the brake pedal do anything.
And then you don't tell drivers about this change.
You can totally see how, right in the lead-up to a fatal accident, a driver is going to be focused solely on stomping on that brake pedal in increasing panic, wondering right up to the moment of death why that's not doing anything. They won't consider the cruise control off button because it's not their most immediate need (braking is), and they've never needed to use it before.
Absolutely reasonable to ask what Boeing could have done better. I'm just not sure the information about Boeing's actions contained in the article gets me all the way to "indefensible", paraphrasing another comment.
Imagine that the failure that caused the nose-down wasn't a failed AOA sensor giving bad readings to MCAS, but some other reason that _also_ wouldn't have been solved by pulling back on the yoke, but would have been by completing some later step on the checklist. Suppose the pilots didn't follow the checklist. Would that be enough information to say that Boeing is responsible?
To be sure, in this case the pilots may have followed the checklist! It may well be the case that Boeing is completely responsible! The checklist items might not have worked, or there may have been a good reason that the pilots didn't follow it, or the checklist might have been crazy, or they might not have had time to do what needed to be done, etc. There's still a whole lot that's not known (or hasn't been released) about what happened.
I'm just not sure that the current evidence, _viz_ that Boeing made an internal software change, that they didn't explicitly call it out to pilots, and that there's no difference in the actions prescribed in the event of an uncommanded nose-down pre- and post-change, is enough to say that the fault is entirely Boeing's for this accident.
And you are writing this after many, many accidents that root cause was pilot not exactly knowing what or why something happens with a plane or plane autopilot (ie. AF 447)
Being apologetic of cost cutting on safety issues is dumb as it erodes culture of safety and encourages others to skimp on safety.
The pilots did not train for a specific root cause of a fault. They trained for a symptom (uncommanded nose down), and the procedure for that situation was unchanged.
> extra burden to quickly determine if something else is wrong
This isn't an extra burden. Pilots aren't doing root cause analysis for failures while they are responding to them. They are trained to try actions in a specific order until something works. It's not like the checklist used to have one action on it and now it has two - the solution in this case was a standard action on the standard checklist.
Pilots do not look around, say "ah, electrical short in elevator actuator" or "ah, bad angle of attack sensor" and then take a single action.
Unexpected situations are always extra cognitive burden.
From my perspective, it is a human factors issue that Boeing failed to consider.
Yes, the emergency procedure remained the same, but even well-trained pilots are still people. And changing the behavior of the system in such situations was not well-advised.
Firstly, there is no single equivalent to "slamming on the brakes" for uncommanded nose-down. This could be caused by a variety of faults, and pilots are trained to respond in a fashion that will be effective for even those in which the first thing to try doesn't work. There is a standard procedure in place to use in this type of situation - arguing that the pilots should only be expected to do one thing with increasing desperation is essentially arguing that they will not be able to respond to a whole host of emergencies causing uncommanded nose-down.
Second,
> Information from the flight data recorder shows that the plane’s nose was pitched down more than two dozen times during the brief flight, resisting efforts by the pilots to keep it flying level ... The standard checklist for dealing with that sort of emergency on the previous version of the 737 focuses on flipping the stabilizer trim cutout switches and using the manual wheels to adjust the stabilizers. [emph mine]
your argument essentially hinges on the assumption that pulling hard back on the stick is a sufficient solution for all the problems that may happen with a plane with the exception of a fault with MCAS (the new system). I don't think that is accurate. Even prior to the the 737 max! It sounds there were a lot of things that would require further action that pulling back on the stick.
There's clearly a problem on Boeing's end here.
> they developed an unconscious/intuitive mental model of the plane
If the plane has a fault, it's frequently not going to behave like their unconscious model says it should. In the happy case, rely on your intuition. In the unhappy case, when things are working as they should, follow the emergency procedure.
Sorr, but this is a little like saying "follow standard procedure to put the landing gear down. If that procedure doesn't work, repeat your intuitive action over and over until hopefully the landing gear goes down." No - follow the emergency procedure in place for landing gear fails to descend.
> not communicated to them for cost-cutting reasons
Take it up with the FAA and the airlines?
Sure. It's not just Boeings fault. I believe the FAA will course correct from this and probably all changes to flight controls will need to be reported to pilots. More lessons written in blood.
I haven't seen any evidence of this. It's definitely not supported by this NYTimes article.
Do you have a source?
"Older 737s had another way of addressing certain problems with the stabilizers: Pulling back on the yoke, or control column, one of which sits immediately in front of both the captain and the first officer, would cut off electronic control of the stabilizers, allowing the pilots to control them manually.
That feature was disabled on the Max when M.C.A.S. was activated — another change that pilots were unlikely to have been aware of. After the crash, Boeing told airlines that when M.C.A.S. is activated, as it appeared to have been on the Lion Air flight, pulling back on the control column will not stop so-called stabilizer runaway."
If the above is true, it's near criminal that Boeing didn't notify pilots about the change. It's also not just about following checklists but understanding the systems of the aircraft.
The more "piloting" computers are doing, it seems appropriate that they will be increasingly, properly, accused of the equivalent of "pilot error". And yet here is Boeing, taking more and more piloting responsibility, while still blaming the pilots when that doesn't work out. It's a deification of engineering: when things go properly and safely, praise engineering; when things fail, blame the pilots.
Actually in the "old" version, there is.
From TFA:
Older 737s had another way of addressing certain problems with the stabilizers: Pulling back on the yoke, or control column, one of which sits immediately in front of both the captain and the first officer, would cut off electronic control of the stabilizers, allowing the pilots to control them manually.
Which makes the car analogy apt.
EDIT: ... Presumably because it's literally the second thing on the list of things to do in the case of a runaway stabilizer.
12 minutes, in this case.
I don't read it that way. To me, it's specifically based on removal (without notice to pilots) of a function that would absolutely have terminated a trim runaway condition.
In other words, Boeing eliminated one well-known and trained-on trim runaway fix, and didn't tell the pilots. This is different from saying that stick pullback is a universal solution.
Its extremely unfortunate, but these pilots had ~10 minutes of flight time between their request to return to the airport noting aircraft control issues, and their crash, during which they completely failed to follow their checklists. Lion Air put those under-trained (if we're being generous) pilots into that cockpit and they are the only ones who carry responsibility for this crash.
Its anecdotal, but most of my older flight instructors have been lamenting the loss of critical pilot skills in the industry, and this appears to be just another example of that.
but still, instead of going trough the full checklist they stopped and tried repeatedly whatever fixed the issue in the past/on the simulator.
a more complete analogy:
"follow these ten step do diagnose a bug on the software"
"but last time it was just a compilation flag, I'll check the compilation flags"
"bug persists"
"last time it was just a compilation flag, I'll check compilation flags"
"bug persists"
"last time it was just a compilation flag, I'll check the compilation flags"
"system halts"
That's one heck of an analogy! Brutal
1. Control column ............................. Hold firmly
2. Autopilot (if engaged) ..................... Disengage
3. IF the runaway stops:
------------------------ [done]
4. IF the runaway continues:
STAB TRIM CUTOUT switches (both) ...... CUTOUT
IF the runaway continues:
Stabilizer trim wheel ............ Grasp and hold
EDIT:
It's #4 that's of interest here. People saying that the interface changed are saying that it's fine if pilots stop after #1, even when dealing with runaway stabilizer for 12 minutes.
This has been discussed in great detail on other flight forums.
Disregard the text on the video description as it's blatantly wrong (describing that the trim tabs still move without the wheels moving; wrong on two accounts: the trim doesn't move without the wheels moving and the 737 uses a jackscrew for horizontal stabilizer trim rather than trim tabs [which is why the pilot can't simply override the aerodynamic force as they could with a trim tab])
Jackscrew operation: https://www.youtube.com/watch?v=rxPa9A-k2xY
The 7-3 has balance tabs to make the control forces lighter in the event of a hydraulics failure, but these are not trim tabs in any sense of the word.
Sure, the correct response is to follow the checklist, but should we really rely on pilots always knowing the correct response? Especially when that goes against their previous experience? Not that I am defending these pilots not knowing the checklists, instead I am arguing we should take into account learning from experience rather than theory.
I see Boeing's point, too, but to me it just means both sides are at fault. The pilots are at fault for not following the emergency checklist. Boeing is at fault for abusing rules to slip in a change based on the assumption that pilots never rely on their own understanding of the aircraft, which I'm sure they know to be false. Air safety is all about human factors, and that's a pretty obvious one.
The one thing that seriously surprised me was that an automated system that is able to point the airplane towards the ground is intentionally fed by a single, non-redundant sensor.
Everything else I've read about various safety systems that limit or override the pilot has explanations about how redundant sensors are used. And how the system does switch itself off in case the sensors don't give consistent results.
It sounds like the old flight manual stated one of two possible methods for dealing with runaway stabilizers. Because the second method (hard back on the control) wasn't in the manual, changes to that way weren't taken into account. Hence a non backwards compatible change slipped through.
I'm a bit baffled about how the previous flight had the problem, apparently misdiagnosed but resolved it, and the new crew had the same problem but was unable resolve it. How's it possible that the plane takes corrective action without telling the pilots? It's not like getting close to a stall is a routine occurrence, is it?
Also, given how important the AoA data apparently is, it may not be displayed prominently enough. In clear weather, level flight flying, it should be really obvious if this is drastically wrong.
MCAS is just one of many different systems on a 737 that can automatically adjust the stabilizer. Because there are so many things that could be the cause of the problem, the checklist procedure for any runaway is just to cutout automatic control of the stabilizers.
Here is Boeing official update to all 737 MAX operators after the crash, which boils down to saying "follow the checklist, dummies".
http://www.avioesemusicas.com/wp-content/uploads/2018/10/TBC...
If I understand this right the pilots must have known the plane did adjust stabilizers, but they didn't have a way of finding out why it did it. And per checklist they shouldn't have cared for the why, but just cut-out the automatic control.
Lion Air didn't even have the basic (optional) AOA Disagree alert[1]
[0]https://theaircurrent.com/aviation-safety/southwest-airlines...
[1]https://www.reuters.com/article/us-indonesia-crash-boeing-ao...
Anyone who was "doing it by the book" was not pulling up on the stick.
Now, maybe Boeing was suggesting via side channels that there were alternate ways to solve certain problems and those side channels should qualify as public documentation... but it may have been intuition earned through experience overriding standard procedure.
Normally, I'd say that those who ignored the spec deserve less consideration. However, when that involves giving less consideration to 100+ passengers as well, that changes.
With a normal software library, you might make the decision to cause breakage anyway, even if it inconveniences users of your library. Or you might not, because you believe the inconvenience will be too great, and instead just document the behavior and make it a part of the API.
With an airliner control system, you need to be a bit more careful, since a pilot depending on undocumented behavior may do so in a way that could cost lives if that behavior is changed. Is the pilot correct to depend on that behavior? No. But that's irrelevant when lives are at stake.
https://www.youtube.com/watch?v=3pPRuFHR1co&t=154
You'll notice that it's a loud, physical event, with a very simple solution.
This happened over twenty times in the accident flight, and the pilots never disabled the problematic automatic stabilizer system.
Pilots are pretty unhappy about this M.C.A.S. situation. They're literally expected to fly an aircraft, and not even being told how that aircraft functions. And while the checklist may eventually take care of this, that isn't a substitute for a professional pilot in the cockpit. Just the lack of training/simulated failures for this new system is highly irregular, pilots are used to and expect such training while transitioning to a new major aircraft version.
The biggest drivers here seem to be cost and Boeing's competitiveness, not safety. I think it might be time for the EASA to trust the FAA a little less, at least until they get their house back in order.
"Word on the street", is that Boeing has lost most their safety reputation and most people just do their job and try to not get burned when the planes start to crash. I want to believe most are just blowing of steam, but I don't know..
In a mostly-robust system, different layers catch and defend against the errors of other layers in the system. For a major accident to occur, holes in multiple layers have to line up that day.
In this case we have four holes that lined up that day - a plane model with a possible rare software bug, an aircraft with a faulty sensor feeding bad information to the computers, an airline company with internal culture that continues to fly a specific aircraft that keeps trying to point at the ground, sometimes without even making an attempt at fixing the problem, and finally, on this fatal day, a crew that did not follow the proper procedures even after twenty-three nose down incidents during the flight.
Even without the MACS system present, the last two holes seem like they would bring down an airliner eventually, from one cause or another.
Pilots are rightfully mad about not being told about the MACS system. But it's just one of many systems on a 737 that can trim the stabilizer to point that it can't be flow. That's why the procedure for any stabilizer problem is to disable automatic control of the stabilizer. The training and checklists that the accident pilots had covered this, and previous pilots flying the accident aircraft did this and then had uneventful flights.
By the way - in small airplanes, you can overcome trim with elevator pressure. That's not necessarily the case on a passenger jet; and not only because it's much bigger, but because the trim works differently [1]. I wonder whether that played a role. I must admit that before I read [1], I had assumed that bad trim is something I can overpower, when push comes to shove.
There are plenty of single components on an airliner whose failure can cause a stabilizer trim runaway. Different airliners handle it differently. On a 737 can you can cut out automatic control, and use wheels connected to the stabilizer jackscrew with metal cables. On other airliners, you can cut out automatic control, then switch second electric control system and use it manually. A 737 stabilizer runway isn't an instant thing, and is a loud event in the cockpit.
This all seems to come down to the fact they wanted to avoid having to retrain pilots ($$$), so these automation changes were kept in the dark.
The crew before them dealt with this same problem but they successfully cut out the trim system. They got lucky and they should have been more vocal in expressing the fault outside of just a post-flight note about it.
The fact that the 737 can auto-trim itself beyond manual elevator authority, due to a SINGLE faulty AOA sensor, is mind-boggling and scary.
Given that the FAA made the decision that it was fine to not retrain the pilots, sounds like there's going to have to be someone to regulate the regulator.
Auto-trim beyond the elevator authority is not a problem as the pilots can take manual control of the trim by grabbing the trim wheel (its in a very obvious spot on the 737).
The actual fix is hard as adding another alarm can get tricky from a UX perspective during an emergency. Probably the only “fix” is to reinforce the value of following the checklist.
Now, that wouldn't have directly pointed to what was wrong, but it would have been pretty suggestive.
(I would suggest, though, that having an optional configuration that lacks robustness for a system that can automatically point the plane toward the ground... a really poor choice of options.)
What happened? The pitot tube and drain were clogged. Static port was clear. This turned the airspeed indicator into an altimeter - it was incapable of showing correct airspeed from the moment of blockage.
The cause of the crash is pilot error. The pilot is expected to recognize from other instruments that the airspeed indicator is unreliable, and this is part of training for instrument rating.
If the MCAS in the Lion Air crash made a similar mistake - using a single data point to determine a stall condition. That is an error. It's functionally "pilot error" to have no means of determining if the angle of attack sensor is wrong, and no mechanism for disregarding its data. Further, the corrective action it took, had the flight condition actually been true, sounds excessive. If a human pilot did the exact same thing MCAS did, I expect the human would be blamed - it would be pilot error to so aggressively nose down that you've exchanged a level flight high speed stall (a rare event indeed) for a high speed dive. That is not a competent recovery, in particular that there's apparently no recognition of the danger of high speed dives let alone recovery from them it's probably a really good idea if your stall recovery does not ensue in a dive!
They created a single point of failure that way. Why?
It's not a single point of failure as we think if it - if it starts acting up, you can easily disable the automatic stabilizer system, per the procedures. 737 stabilizer runaways take several seconds to take effect, and are recoverable afterword. Later you can switch flight computers and then be using clean data, though you are supposed to leave the stabilizer system off for the remainder of the flight.
Airbus uses three angle of attack sensors and compares them. They've had at least one crash when two sensors failed in a consistent way.[1] The vulnerability of aircraft flight control systems to bad AOA data is well known.
[1] https://news.aviation-safety.net/2010/09/17/report-blocked-a...
As comparison, Southwest had the indicator but has now also installed an enhanced AOA Disagree indicator as a result of the Lion incident
https://theaircurrent.com/aviation-safety/southwest-airlines...
I guess what really bugged me about it is how un-Boeing-like this behaviour was; a computer overriding a pilot (even if there is a way for a pilot to override it in turn). It's fundamentally an Airbus-esque design.
As I read this article though, everything fell into place. As you read it you start to see, with utter clarity, exactly how this happened organizationally.
It's well known that Airbus uses software flight envelope protection to enable them to reduce the safety margin applied to the airframe, reducing weight. In other words, fuel efficiency is improved by making airframes less airworthy and compensating for it in software. I don't actually disagree with this as such; it's been demonstrated to be a sound approach, but historically Airbus's domain.
Essentially, it seems like what happened here is that Boeing finally felt the need to adopt similar techniques to compete with Airbus on fuel efficiency (though regarding engine size issues, not airframe safety margins, but still making a plane's airworthiness more caveated and fixing it in software). Essentially, we're witnessing the point at which Boeing feels its traditional user interface philosophy (do what the pilot says) is conflicting with market pressures.
If this were a new plane with a new type rating, this wouldn't be unreasonable. Trying to tack this on to an existing plane, and not only that, but doing everything in your power to minimise the amount of transition training, is OTOH extraordinarily egregious.
The problem with this change isn't so much that Boeing's reasoning for not telling pilots about it isn't logical. If anything, the problem is that their reasoning is utterly logical: the checklist will solve the problem anyway, no matter the cause. You can see how this decision must have percolated through different teams at Boeing, through regulators, via this unimpeachable-seeming logic. The market pressures involved (fuel efficiency and retraining costs) would have made it particularly hard to contest. It's a completely logical line of reasoning... yet here we are with fatalities.
I'm very interested to note, though, this new revelation (to me at least) that the yoke behaviour re: extreme deflection mitigating stabilizer runaway was removed in the MAX. So what was Boeing's justification for this change? Was it even mentioned? If not, what on earth were the regulator's justifications for allowing it to go unmentioned? I want to hear those justifications, since it seems impossible to justify. I was under the impression that compatibility of type ratings fundamentally revolved around an absence of differences in how two planes handle, and how they respond to the yoke.
I should add, the reliance on a single sensor is also remarkable; makes me wonder if this entire subsystem was really rushed and not given proper design review, which would make sense given the circumstances (panicking to get a product to market).