FAA Issues Boeing 737 AOA Directive After Lion Air Crash | Hacker News Reader

FAA Issues Boeing 737 AOA Directive After Lion Air Crash

aviationtoday.com

81 pointshef1989866 comments

I think one of the bigger issues that isn't being addressed is the current up and coming pilots don't have the basic pilot skills to overcome the systems when they fail. Flying has become so automated, that a pilot is only required to fly on take off and landing, and even landings are optional on a lot of planes now.

When one is confronted with a lack of control caused by those systems, and having been so removed from general flying that they end up panicking and crashing.

It's such a deal that the FAA is reviewing flight training because Airline pilots are struggling with even the basics such as go arounds and stall recovery.

Below is an article about the FAA's struggles[1]. And an article about how most pilots couldn't actually pull off a Sully type landing[2].

https://www.flightglobal.com/news/articles/industry-sounds-w...

https://nymag.com/news/features/53788/index1.html

This is somewhat being compounded/driven by the pilot shortage/crunch in the aviation industry. Typical costs for becoming a pilot are high and the immediate returns are low. Sure if you become a senior Captain with a major airline like United you can make good money, but before you get there you start by flying for a regional making as low as $20k a year:

https://www.economist.com/gulliver/2016/03/09/america-is-run...

And if you are facing student loans that is a wringer.

So now that airlines are expanding, the majors are having trouble finding experienced or qualified pilots since many people forgo pursuing it as a career.

For high growth markets, like Asia, they have resorted to lowering license requirements with things like the Multi-crew Pilot License (MPL) which is all sorts of alarming (remember when that Asiana flight crashed on landing in SFO?).

Hence why I am a certified commercial pilot, Embry-Riddle graduate, and am now a software developer browsing HN.

Not looking back.

Also I understand pilots are discouraged from taking risk when flying a small private plane on their free time as it could hurt their professional license. But on the other hands that’s the sort of experience that becomes handy in those catastrophic failure scenarios.

Quite the mis-aligned incentive. At the very least hard core simulator time to initially train the reflexes and some real 'stunt' flight hours in actual air to overcome the fear when real forces are involved seems to be in order.

Yes, there's a whole part of training dedicated to Threat and Error Management (TEMs).

We don't take risks, but we do generally train for many common failure scenarios in a given plane (everything from incorrectly reading / failed instruments to equipment failures and fires etc) when getting endorsed / type-rated for it.

How does this work? I can see how it might look bad if they had an incident bad enough to report to the FAA. But what kind of "risky" flying is safe enough that pilots should try it, yet runs a substantial risk of having reportable incidents?

Practicing very slow landings, or power-assisted landings, would fall into this category. There's a wide range of marginally legal behavior such as low flight, back country flying, or even gliding, that might serve a stick-and-rudder pilot well yet if a concern is raised either because of an accident or because of an ATC or pedestrian concern would result in action on a pilot's license and subsequent additional explaining to airline HR.

I'm not sure that's a thing - PPL holders are supposed to be able to demonstrate flapless, partial power and glide (no power) approaches; plus low-level navigation. CPLs do more advanced (harder) versions of the same - it's part of most training curriculums and is tested on during exams.

I know these are all facts, and used to have empathy for my neighbor across the street who is a pilot for one of the majors.

Then one night at a neighborhood party he was complaining about his pay and let it slip that he had to take a pay cut that was more than some people's entire annual take-home.

I don't feel bad for him anymore.

That sounds more like a surplus than a shortage.

It's both - a surplus of pilots who are early in the pipeline, and a shortage of those who are experienced enough to be hired by a major carrier. When the training period is long like this, supply always lags demand significantly. If you lower wages during a surplus, you're setting the stage for a shortage a few years from now.

Edit: Also keep in mind that a "shortage" claimed by an employer can sometimes be nothing more than a shortage of people willing to work for the price they're paying. ;)

How does a lack of pilots sound like a surplus?

See sibling reply[1]. If you can't make much as a regional pilot, that sounds like supply swamping the demand. But if you need all those hours to be a different kind of pilot, that may translate into a shortage of that kind.

[1] https://news.ycombinator.com/item?id=18410665

It's more like a sticky market effect than it is a surplus of supply. When I worked in the aviation industry 5 years ago the problem was education costs had increased rapidly while entry-level wages have largely stayed the same. This is the same trend as the overall market, but in aviation the cost-wage differential was especially large. Airlines began to see the the number of college-age, young students entering the pilot industry start to fall. Additionally, the pipeline for new private-sector pilots started to shift: fewer students entered high cost private flight schools while more enlisted in the Air Force which offered better incentives and (more or less) free training at the cost of a 4 year military commitment.

This squeeze led to majors upping their compensation for tenured employees in order to encourage them to stay. This is why Senior Captains have seen increases to their average compensation while entry-level pilots at regionals haven't seen much change (regionals often operate on shoestring margins). This, in the short term works, and even now it's part of the reason that Pilots at majors end up having longer careers at the same carrier than average private sector industries. The benefits in scheduling and pay that they accrue the longer they are at a single airline are some of the strongest employee retention benefits of any industry today. Perhaps not as strong as the pension benefits of the 50's but certainly stronger than Software Engineers which have seen decreasing average employment duration in recent years.

This means that while potential new pilots have been choosing alternative carreers and older pilots stay longer, the age distribution of pilots has become unevenly distributed. There are a lot of older-aged pilots being paid a lot, but there have been overall less pilots entering the industry.

But here's the catch: Airline pilots are one of the only professions which have federal regulations governing age eligibility. That is, pilots must retire at age 65 according to the FAA. This age requirement was increased from 60 back as recently as 2007:

http://www.washingtonpost.com/wp-dyn/content/article/2007/01...

But the FAA added a qualification where international captains over 60 had to be accompanied by a co-pilot under 60.

https://www.airspacemag.com/daily-planet/too-old-fly-1809623...

Point is there is an aging pilot population which, literally, will age out of the profession and not enough younger pilots to fill the gap.

At any rate, the pilot crunch is a really interesting market situation and I haven't been in the industry for 3+ years so my perspective is likely dated.

Classic talk on this topic: Children of Magenta - https://www.youtube.com/watch?v=pN41LvuSz10

99PI - https://99percentinvisible.org/episode/children-of-the-magen...

This talk really opened my eyes to the quality of pilot training and effects of automation dependency. The fact that this was a problem in 1997 and that systems have gotten significantly more powerful and complex since then.

Anyone interested in self driving cars should watch this too.

Agreed. AoA indicators are extremely useful but until recently most pilots learned to fly without them, and they still should. The loss of an AoA indicator absolutely should not prevent a skilled pilot from flying the plane.

Losing an AoA indicator entirely is probably easier to deal with than false readings from one, but again, a skilled pilot should be able to tell when the readings are false and fly the plane.

Looking closely at what they said, it seems like the AOA sensor was automatically making trim inputs which caused the plane to ultimately develop a very nose-down trim condition. Basically a high tech version of a runaway trim control which is why you often see split trim buttons so a switch failure can't cause a runaway.

You're right and this point needs to be emphasized.

Unless you know how to turn off the Flight Control Computers and the right moment to do so - which may or may not even be in the checklist (any pilot want to chime in?) - then even with autopilot disengaged modern jets will still operate flight control surfaces if a 'dangerous' configuration is detected.

Of course with bad data going in you get bad data out, so the FCC can take actions that put the aircraft in danger as it thinks it's in a situation is isn't really in.

As far as I understand it, you can't simply override these inputs with pilot input, you need to actually shut down the FCC to get back control.

https://leehamnews.com/2018/11/07/boeing-issues-737-operatio...

Yes, in fact the AD update to the flight manual is a list of further indications for when the Runaway Stabilizer procedure should be followed.

Is not about flying without AOA indicators. It's about the flight control systems incorrectly reacting to a bad AOA sensor reading and trimming down the aircraft stabilizer.

Near where I am based in China there is a mandatory pilot training facility for one of the major airlines. Pilots must train for one week every year in the simulator. Once every month or two I run in to foreign pilots in town for this purpose. I met two recently and specifically asked them about the Lion Air event and their training for failures, based on the assumption that they must train for complete system failure. They said that in truth the probability of complete system failure is so low that completely manual training is now considered irrelevant to annual review and is not even performed. As an example, they said that the rudder of a typical plane has three redundant hydraulic systems, all of which are routed differently through the airframe and all of which would need to fail to lose control. In short, they have statistically more significant areas to train on, limited training time and budget, and it has been correspondingly deprioritized. This seems to be in direct contrast to boat sailing, where one of the most common and significant drills given to new sailors right across the world is learning to effectively steer a vessel using sail trim to compensate for a loss of rudder or rudder control.

You might be right, but this case is not the best example. Here loss of AoA sensors led to plane trimming down when flying in manual mode. So, more manual flying worsened the situation here (because they may not have cutout the stab trim...)

I'm curious what the difference is in feel between a 737 and the GA trainers they all started on. I've flown without AoA (in the clouds no less) or even a stall horn but without seeing anything you still know what the plane is trying to do aerodynamically.

At low speed most planes will get a little "rough" (kinda like high frequency, low amplitude turbulence) and the control surfaces will respond sluggishly whereas at high speed the controls feel sharp, trim controls have vastly more effect and slight changes in direction have a stronger feeling of G-force. I haven't flown anything with hydraulic or fly-by-wire controls so I'm not sure how much logic is being applied to augment input depending on speed which might have an effect. That and heavier airplanes respond more slowly and generally dampen aerodynamic oscillation more.

What happens leading up to and in a stall is very heavily dependent on the plane. A Cessna 152/172 and Diamond DA-20/40 have very different stall characteristics despite being similar size. My 310R and my old Mooney are very different and I do not think I ever fully stalled the former. I know for sure I never fully stalled any straight-wing jet I fly and have no direct experience with swept wing jets (like the one in question here).

So, stall characteristics just don’t generalize well in my experience.

Sure, a cessna 152/172/182 or piper will universally mush then break somewhat cleanly but evenly downwards where as most aerobatic planes/taildraggers will try to enter a full spin if you look at them funny. I'm more interested in how you can get into a high AOA, low airspeed condition without noticing something is off. As AOA increases, indicated airspeed decreases further so it seems like nobody was even bothered so much as to glance down at it for a split second during the lead up. Also, how did they allow themselves to get so much pitch or so little throttle that a full stall developed.

Edit: Re read and realized they never actually mentioned a stall, but rather they probably thought they were approaching one and just kept mashing the nose down trim switch because AOA was reading high so they probably over-sped the aircraft. I guess my point becomes: why does it seems like they were only flying off of AOA??

Edit Edit: The wording suggests it wasn't even the pilots making the input, the AOA sensor was making automatic inputs and the airplane probably over-sped and broke up (or someone freaked out and pulled up hard).

> I'm more interested in how you can get into a high AOA, low airspeed condition without noticing something is off. As AOA increases, indicated airspeed decreases further so it seems like nobody was even bothered so much as to glance down at it for a split second during the lead up.

Maybe you were confusing this accident with AF447, which indeed stalled due to an excessive AoA?

This is an interesting take, and one I'm surprised about given the recent increase in required flight time for an ATP (Air Transport Pilot) certificate from 250 hours to 1500 hours. I'm sure I'm misunderstanding, but wouldn't current up and coming pilots be more experienced to handle these types of situations given the additional hours needed?

There is a semi-famous write up about this: http://www.bainbrdg.demon.co.uk/Papers/Ironies.html

An earlier article called out poor maintenance at Lion. Here we have:

>... airspeed indicator had been malfunctioning on four consecutive flights prior to the crash ...

As a non-pilot, I'm surprised that a malfunctioning trim system could cause a crash unless it happened close to ground. When I hear "trim", I'm imagining minor control surface adjustments that the pilot can easily override with his control stick.

Does it take significant effort to overcome trim? Or does a bad AOA sensor add to pilot confusing during recovery?

On a 737, trim adjusts the pitch of the entire horizontal stabilizer. As such, in the extreme, it actually has more control over pitch than the elevator (which is controlled by the pilot's yoke). If the trim runs out to the limit, and the pilot doesn't correct it, the elevator does not have enough authority to override the trim.

Do you have a source for this claim? Two different comments in this thread are stating opposite claims

https://news.ycombinator.com/item?id=18409562

This explains a bit about how the THS works: https://www.skybrary.aero/index.php/Trimmable_Horizontal_Sta...

The Rostov-on-Don crash in 2016 appears to have been caused by downwards trimming for 12 seconds. This caused a -1g dive from which the pilots were unable to recover. It's inconceivable they wouldn't have been attempting to pull up with the elevator. You can't not notice -1g. https://en.wikipedia.org/wiki/Flydubai_Flight_981

Now the final report is not out yet, so it's not clear if the downwards trim was somehow manual (which doesn't really make any sense), or a similar problem to the Lion Air crash. Either way, it's worried Boeing and the FAA enough to issue this AD.

Wow. Didn't know this. I can understand why they do it this way--it's probably the lowest drag solution to the problem. But it sure swaps my connotation of what "trim" and "elevator" mean.

There's a common GA plane (M20) that moves the entire tail for trim, both the horizontal and vertical stabilizer as one piece.

Basically conventional (not a canard) stable (not an F-16) aircraft have the CG in front of the center of lift of the main wing so it wants to nose down. The tail then acts as a counter balance, pushing down to keep the nose up. Because airspeed over the smaller surface effects its downforce and the center of thrust is not always in line with the center of drag, they add trim to prevent the pilot from having to maintain constant control pressure. Some do it by biasing the control surface itself, either in the control mechanism or with a tab that aerodynamically moves the surface, or by changing the angle of attack of the horizontal stabilizer.

I’m no pilot, but one of the first things you learn in Kerbal Space Program is that CoG behind CoL is a completely unflyable configuration.

Ha, yea. I was deliberately using that terminology hoping people might be able to draw some parallels with their KSP experience.

Is the trim setting (pardon my lack of knowledge) easily visible to the pilot?

In a Boeing, absolutely. The designers know it is important, and they devote a slice of cockpit space to a comically large "trim wheel." It has a white stripe to highlight automatic movements, and a large handle to allow manual control.

See: https://www.quora.com/What-are-the-spinning-wheels-in-an-air...

Recent life-threatening runaway trim incident, on a modern jet:

https://50skyshades.com/news/business-aviation/bea-pilots-sk...

Here's one example where loss of an elevator trim tab caused the pilot to experience +17G, leading to incapacitation/unconsciousness and a fatal crash: https://www.ntsb.gov/investigations/AccidentReports/Reports/...

(also a non-pilot, I found the above surprising, but there it is..)

The plane in question was a highly-modified P-51 taking part in the Reno air races. Looking through the report it was routinely pulling 4G in the turns already (at ground speeds of 360-380kts) so quadrupling that in the face of structural failure isn't beyond belief.

It was pulling 4G turns by using the elevators. My surprise was that losing such a tiny piece as a single trim tab would cause a 17G pitchup.

But yes, that was a highly modified plane pushed way past any sensible operating limits, so maybe one shouldn't generalize too much from it.

This is where the NTSB report most directly addresses the issue: "...the loss of the left trim tab’s downward force on the elevator resulted in a concurrent sudden upward deflection of the elevator, a sudden and forceful aft movement of the control stick, the pitch-up of the airplane, and the rapid increase in G."

At this point, the trim tab was still attached, but its link assembly had broken.

In addition, the report notes "two major modifications to the elevators each made the airplane more sensitive in pitch control: the substantially increased elevator counterweights (about twice that of a stock P-51D) that overbalanced the elevators and the substantially decreased elevator inertia weight (less than half that of stock)."

Just from top of my head, a faulty Airspeed Sensor and some quirky AOA sensor behaviour was one of the reasons why the Air France flight crashed into the Atlantic a couple of years ago. These two factors contributed to a loss of situational awareness, a lack of system knowledge, the quirky AOA sensor behaviour, meant the pilots never regained situational awareness.

And trim may sound trivial, but sudden changes of the center of gravity can be instantly fatal. Once you leave the space of small trim adjustments needed, for e.g. fuel consumption, recovery may well be impossible. That's why load planning of planes and cargo handling / stowing are so important. You don' t want to end up with a center of gravity in a place that is incompatible with flight.

From memory; pitot tubes froze over and airspeed became unreliable. Autopilot disengaged and they had a stall warning. One pilot went full nose down, the other wanted full nose up. The latter probably believed the flight envelope protections were still in place which would not let you stall even if you did full nose up. But they weren't because the flight computer switched to alternate law when airspeed was unreliable so they had full control of the airplane and (probably) didn't realize it. Additionally when presented with two different control stick inputs the flight computer just averaged them, and since there was no force feedback there is no tactile sense that you're arguing with the other pilot over what you should be doing. Also only one person is supposed to be in command.

My memory is also that the senior captain was in the third seat and didn't have a full grasp of the situation either.

I did read the air accident report quite a while ago so this might be wrong, but in summary, it's not exactly arguable that the aircraft did the wrong thing (it did as designed minus the freeze issue which is suboptimal but something that can happen) and the pilots didn't exactly do the wrong thing either since they may have little to no experience flying the plane in alternate law (certainly not at high altitude and speed), and it's not unreasonable to be fully aware that they had full control (as opposed to the flight computer mediating and not letting them do something stupid like stall it all the way down to the ocean).

Lastly, when in cruise it's not completely trivial to fly like a Cessna at 1,000ft. At cruise on a big heavy jet you can't go much higher as the air is too thin to fly. You can't go much faster without braking the sound barrier. You can't go much slower as you'll stall. I believe it's called "coffin corner", and while commercial passenger planes do it every day it's not quite as simple as you might think.

There's no excuse for secretly averaging widely divergent inputs, especially in alternate law (which the plane wouldn't be in unless something is going wrong). It should have picked one and indicated to the other pilot that his/her input was being ignored. IIRC this isn't an issue on the Boeing design, only Airbus.

One the contributing factors was that the stall warning cut out when the nose was pointed higher up, giving the copilot the impression his attempt to full nose up was the right decision. When the nose lowered the stall warning kicked back in. IIRC this was because the computer saw the full nose up as an "impossible" condition and decided not to issue the stall warning. As the nose lowered certain values came back in-range so it issued the warning.

Without these two wildly incorrect behaviors the plane probably wouldn't have crashed.

It didn't secretly average inputs. The plane gave alerts both visual and audible (DUAL INPUT) that the pilots were providing conflicting inputs. I agree that the force feedback of the Boeing FBW system is better, but the pilots should have understood the alerts provided, and either communicated or made use of their override switches.

Solving the problem by ignoring one of the pilots would be an interesting engineering problem. How do you choose?

it's not exactly arguable that the aircraft did the wrong thing

I still think it's arguable that the "right thing" is for the plane to say "Hey, I don't know how to fly this thing anymore, so I'm giving up! Oh and since I'm so confused, the plane is in alternate law, so all of those nomal-law protections you usually have aren't there." and then handing over control to the pilot that hasn't been actively flying the plane and expecting him to suddenly figure out what's been happening and why the autopilot gave up.

Since the pilots did pretty much the opposite of what they should have done, maybe the plane could have done better itself, even with the unreliable airspeed?

> the pilots didn't exactly do the wrong thing

Wasn't the one pilot pulling back fully on the stick and not communicating that he was doing that? And weren't the pilots aware that they were in a stall scenario? Which means that pulling back fully was the exact wrong thing to do?

My impression is basically that it was almost entirely the fault of the pilot that was pulling back the whole time. Of course, it sounds like Airbus' controls could provide _a lot_ more feedback to pilots, especially when there are divergent inputs.

All cintrolls, invl. Boeing, are fly by wire now. And yes, pulling back was exactly the wrong thing. I would have to look up the rport again, but if I remember well pulling up silenced the stall warning. So nose down -> stall warning, nose up -> no stall warning.

If memory serves me well, Air France revised their training afterwards to cover that exacz scenario, that pulling up silenced the stall warning was expected system behaviour. Complex systems require in depth system knowledge and quick reaction times. That is one of the points I took away from that incident.

> flight computer switched to alternate law when airspeed was unreliable so they had full control of the airplane

Alternate law still enforces bank angle protection. With full control (direct law) it could have entered spin. (or maybe they could realized situation earlier)

Stall warning was confusing because pulling up even more removed warning temporarily because extreme AoA was interpreted as sensor failure.

IIRC they hit the ocean so fast, that death must have been instantenous.

Also IIRC the pilots didn't know what's going on until they hit the ocean.

The major difference is that the Lion Air crash took place during the day, in clear weather. AF447 crashed at night, in poor weather conditions.

AF447 was a bad airspeed sensor and a switch of the control system from "normal law" to "alternate law" as a response to the momentary bad inputs. https://en.wikipedia.org/wiki/Air_France_Flight_447

An uncorrected out of trim condition can be quickly fatal. On the light planes (piston and jet) I fly the arm force required to overcome an out of trim condition is considerable even for a modest amount out of trim. A failure of the electric trim control is a highly acute emergency and you should know at least two ways of immediately killing that system.

Countering trim forces was the first scary thing I learned about go-arounds, the first time I did one, my instructor let me figure it out all on my own, and battling heavy nose down trim when you're trying to make best climb is stressful in a training aircraft in a non emergency, I can't imagine what the forces are like in a large aircraft in an emergency.

I know nothing in this space, but why can't the AoA sensors be fixed / replaced to not generate the bad data that apparently is causing the FCC to generate the Jose down conditions? It seems strange that it sounds like a hardware problem thats being addressed by changing text in a manual. To me that sounds like telling a driver with a flat tire to just steer opposite of it so the car stays straight instead of replacing the tire.

An erroneous Angle Of Attack sensor would not be the cause of a crash (though stated as so in this article) - it could certainly be a contributing factor. The aircraft, from the public evidence so far, was still flyable.

If I'm reading this correctly, that sensor fed into computer which was automatically controlling trim settings. If that's the case, an erroneous reading could change how the plane flies dramatically.

IANAP ... please correct me if I'm wrong here.

The trim system on this aircraft is not stronger than the input the pilot can make via the pitch control system (the yoke and its trim override switch)

Edit: clarified about switch

Is that an and or an or? Would the yoke have avoided the crash by itself with enough applied force, or did the pilots also need to determine that the aircraft was trimming itself incorrectly and do something special to disconnect that? Seems like they didn't do that, in either case.

As usual, for something devastating like that to happen you need more than one thing to happen or go wrong at the time nowadays.

Reply on news.ycombinator.com