Hacker News Reader

Comment by Animats | Hacker News Reader

There have been a surprising number of air disasters in recent years caused primarily by air data sensors returning false data.

- Birgenair Flight 301 - B757. Pitot tube clogged possibly by insect nest, false overspeed indication, autopilot commanded pitch-up, alarms, stall warning, crew confused about speed, loss of control. 189 dead.

- Air France Flight 447 - Airbus A330. Well known. Pitot tube clogged by ice, confusion about airspeed, loss of control. 228 dead.

- Saratov Flight 703 - An-198. Pitot tube frozen. Three airspeed indicators all disagreed. Loss of control. 71 dead.

- Lion Air - Angle of attack vane failure, from parent article.

That information could be checked against GPS, and at least one aircraft does this. But that has its own problems.[1] Checking against an inertial system is another possibility. Those are complicated, though. The classic airspeed, altimeter, and angle of attack vane are so simple.

[1] https://www.gpsworld.com/gps-disruption-a-full-fledged-aviat...

View on HN

So, all it takes is one malfunctioning sensor, which luckily does not happen very often. Add an autopilot which slavishly follows the faulty sensor, as only computers can do so well. Plus two confused pilots, trained to rely on autopilots, suddenly required to do some serious debugging of a complex multi-sensor, multi-control system in seconds, under extreme stress. To sum it all up, you are dead.

I contend that the primary cause is not the sensor but the overconfident reliance on autopilots. When an autopilot intervenes, its whole purpose is to resume stable flight. Should anything unacceptable happen instead, the autopilot ought to switch itself off. Too many (blind) cooks spoil the broth.

Actually, I am shocked at how sloppy their approach to reliability engineering must be not to have even thought of such a basic thing. Reminds of that NY finance company that lost half a billion $ in half an hour because they could not turn off their buggy automatic trading system.

You are nowhere near qualified to say anything about shoddyness both from an individual standpoint and statistical. The competency of aircraft engineering is the highest in the world it cant even be compared to your average financial engineer. To make flying complex systems like planes statistically safer then driving a car is an incredible achievement of engineering.

Numbers show that as Automation has gone up and crashes have gone down. It doesnt establish causation but correlation is a supporting data point.

Additionally, in many of these systems, autopilot does give back control and the pilot in a panic proceeds to royally screw up. The France flight 447 is a specific example of this.

The best way to handle this is to not only train the pilots differently but to change the autopilot. Multiple redundant sensors using different ways to measure airspeed so the detection of a faulty sensor is possible through automation.

Pilots and machines are nowhere near perfect. The goal is not to put both systems at odds, but to move both towards perfection.

I will concede the point that I am not qualified in aircraft engineering, just in autonomous systems, which is a kind of meeting ground here. I have seen way too many people getting over enthusiastic about the capabilities of autonomous systems of all kinds, which I think is a real danger here. It is strange you quote Air France 447 as an example of some kind of superiority of automation, when in fact its problem was caused by eliminating physical force feedback between the pilots by (over) automation.

This is in no way to detract from your analysis, but Air France wasn't instrument error, it was really pilot error.

The instrument malfunctioned, but one pilot was pulling a lever up, while another pilot was pulling another lever down. This generated a stall. It could have been avoided if the cabin was more well lit, so that the pilot making the error could have been visually seen to be making the mistake.

I maintain this is a flaw in the Airbus UX. When they get conflicting inputs they silently average them.

Boeing uses force feedback so the two pilots are aware that something is counter-acting their input.

I really don’t think the Airbus strategy is defensible - certainly not when the two inputs are far apart!

I agree: there should have been a warning that both pilots are doing (very) different commands. Airbus is responsible here.

Boeing is responsible in the article too: adding such new feature without properly communicating with the pilots??

I have a hard time believing the statement from the fine article that the presence of the MCAS system was unknown to the pilots. I'm not a pilot, just an interested spaceflight and aircraft enthusiast, and I know about that system. There is no way that the pilots did not.

Maybe the pilots did not receive proper sim time on a 737MAX, which I believe has different enough instrumentation and automation systems to require a separate sim from the standard 737. But they almost certainly knew which systems were on the plane, and how to operate them under normal conditions.

>Maybe the pilots did not receive proper sim time on a 737MAX, which I believe has different enough instrumentation and automation systems to require a separate sim from the standard 737

Isn't part of Boeing's marketing strategy that the 737MAX actually DOESN'T require additional training? The entire point of MCAS is to make the larger 737MAX react to control inputs just like a smaller 737, so flight crew can just fly them all "like a 737".

There were too many warnings already. That was the problem. More warning could not have helped. Instead, the fact that the pitot tube was inoperable should not have led to so many alarms, just one, and not an urgent one since the autopilot could have kept the throttle at its current setting. (Air speed is independent of land speed, and it's much more important to know the airspeed than the land speed, but wind speed is not going to fluctuate so much, so land speed can be used as a proxy for airspeed in that situation, and anyways there wasn't anything to be done about the lack of airspeed indication in that situation.)

This narrative is incorrect. The aircraft gave audio and visual warnings when there were multiple control inputs. And we can infer that the crew on AF447 noticed them because the 2 (IIRC) instances of dual input lasted less than a couple of seconds.

And the captain actually noticed it, told the pilot pulling back on the stick to push forward, who did momentarily and then he pulled back again.

If you read the whole transcript and report it really does show just how wrong the pilot pulling back got it. And it speaks to the training and understanding that pilot had of how to fly the electronics vs how to fundamentally fly an aircraft.

Agreed. Also, I can't imagine the feeling of the captain who was asleep in the crew area until the very end.

He quite literally kicked down the door to the cockpit and asked "What the hell are you doing?"

Moments later, the co-pilot at fault said "but I've had that stick down the whole time!"

Indeed; it was a stunning lack of basic airmanship; I would expect more from a 10-hour pre-solo student pilot...

I'd expect it from a student, because they're operating consciously. I'm not sure I'd expect it from someone who is in shock, who's experienced thousands of hours of uneventful operation and suddenly everything has changed.

I don't know easily panicked mentality can be tested for. Training is one thing, but training is typically designed to instill automatic response to predetermined stimuli. Sometimes what you want is a reset back to first principles and activation of conscious reasoning. Which is exactly what you don't get in shock or panic, and is the reason for training automatic response.

To me, this moment is one of the failure points in the incident and is entirely the captain’s fault.

Captain: What the hell are you doing?

Benin: We've lost control of the plane!

Robert: We've totally lost control of the plane. We don't understand at all... We've tried everything

Now, instead of a team working jointly on a problem, we have two juniors feeling that they have to justify and defend themselves to their captain. It’s a really poor psychological position to be in with a time-critical problem to be solved.

It’s a human response, obviously, but I’m betting it’s the opposite of what his training would have recommended.

IMO, in that moment, the Captain either needed to get his crew problem-solving, or take control himself. unfortunately he did neither.

I think purely based on the text, your opinion is correct.

But I think at this point the plane was plummeting belly down (not nose down) to the ocean and they only had like 90 seconds to live.

By that point I think they all knew they were dead, even though their words don't indicate it. They were in pitch black conditions :(

My understanding is that aerodynamic modelling and simulator runs have shown that a committed decision to push nose-down, regain airspeed and recover from the stall, could have been successfully initiated all the way down to 5000 feet. The Captain had another 2mins 15secs (approximately) before reaching that level, and another 30 secs before impact after that.

There was time, if only someone had said the word ‘stall’ out loud.

Not silently. There's a voice alert, "Priority Left" or "Priority Right", and a red light. But with other alarms active, it's likely to be missed.

Airbus statement on sidesticks. [1]

Boeing position on sidesticks.[2]

This is a long-standing controversy. The F-16 fighter was the first aircraft produced with a sidestick. But that was so the pilot could control it during high-G maneuvers. Airbus's use of it has been controversial for years. For a good overview of the Airbus control system, read "Fly by Wire", by William Langewiesche.

But that wasn't the problem with Lion Air.

Airspeed. Attitude. Vertical speed. Altitude. If the pilot is operating on wrong information for any of those, they may lose control of the aircraft. All large aircraft have more than one of each of those. But when they disagree, or fail in a common mode, big trouble.

Whether some other data source like GPS or inertial should be used to back up those is an open question. GPS can and is jammed. Inertial systems drift. Both are complicated. The classic instruments are so simple. But they rely on tubes to holes in the aircraft skin, which might become plugged for a number of reasons - ice, maintenance covers, insects, or dirt.

[1] https://safetyfirst.airbus.com/app/themes/mh_newsdesk/docume...

[2] http://archive.li/x0Pn8

OP says primary cause. Which it was. Pilot error happened after, and because of, instrument error.

Being well lit is one part. But also, if the pilots had been in a Boeing that may have helped as well.

Airbus joystick vs Boeing yoke. The latter would have provided feedback to the pilot not panicking and pulling up incessantly

Of course it is better to have more sensors, but that is not the whole issue. The problem is that firstly, the automation is unable to deal with multiple complex failures. Secondly, pilots are not always capable of dealing with those same failures, particularly when the automation itself is part of the failure. The main solution to this is better training. But we should also question if the automation should include better handling of failures before shunting the problem back to an overwhelmed pilot

I agree with you 100%.

The problem is going to get worse. Right now we have thousands of vehicles with trained highly-regulated operators (airplanes) relying on automation, who can be complacent until the automation fails and then must be alert and able to handle a complex situation in a matter of minutes. Soon, we will have millions of vehicles with barely-trained almost unregulated operators (cars), relying on automation, who can be complacent until the automation fails and then must be alert and able to handle a complex situation in a matter of seconds.

Could the ram air turbine tell you the airspeed? Or the GPS?

The GPS can't account for winds, which can reach hundreds of miles per hour.

GPS can tell you ground speed but not air speed. The latter is the crucial informatiom the pilot needs.

They are different but by how much? Could not ground speed be used as an approximation in an emergency situation? Put the airplane level, put engines on a reasonable power and get to a place where you are not forced to act second by second. When the airplane is level and has a reasonable ground speed should that not preclude stall?

They can be different by a lot - easily more than 100mph in some conditions.

How do you propose to get the plane level without knowing the airspeed, pitch, and/or angle of attack? If the pilots or computers had this information, then they would not need the GPS system you describe.

You are describing a navigation system, not a flight computer or an emergency recovery system.

The ram air turbine is only deployed in irregular or emergency situations, and even then, no it would not be a reliable airspeed indicator unless it was reengineered to be one somehow.

4 accidents over how many years and how many flights? I'd say that was very, very few.

Flight 301 - 1996 Flight 447 - 2009 Flight 703 - 2018 Lion Air - 2019

Flights per year: 40 million just this year

https://www.statista.com/statistics/564769/airline-industry-...

Yeah but almost 400 dead people is still 400 more disasters than it should be. Only us engineers can solve this problem and we have a moral responsibility to solve this problem.

> we have a moral responsibility to solve this problem.

Not everyone (including myself) has the same morals as you.

I'm not claiming there is no trade-off saving 400 more people (as basically all everyone replied to me) but engineers have an implicit agreement with the society and ethical practice is a big part of engineering, at least this is what I was taught. The reason this trade-off is irrelevant is because if you're not employed by an aeroapace company, you do not have any implicit responsiblity regarding those 400 people. So what is being claimed here is not that we all should stop and save those 400 people. E.g. I work for a telematics company I'm absolutely responsible of every person injured in a car crash that my engineering solutions could save them. But the same cannot be said about casualties in airplane crashes.

I used to work for Boeing on the 757 stabilizer trim system. I can verify that the engineers who work on such flight critical systems are very much aware of their responsibilities to make them as safe as they can. None of them want anything to do with creating an unsafe design.

Or perhaps our engineering skills, time and resources would be better used saving several order of magnitude more people from death from domestic accidents or drowning.

This sounds like a process, not an engineering problem. The plane had known faulty sensors, and the pilots didn't know about crucial operating features of the plane.

No amount of engineering will save you from this, if you add more they'll maintain it less so that'll fail instead.

The pilots, as part of the 737 type rating requirements, were surely exposed to the QRH checklist for runaway stabilizer, the first page of which is here: http://static-s3.sokoloff.com/flying/other/other/737-QRH-Run...

The Emergency Airworthiness Directive issued after Lion Air 610 crash was, in summary, "Pilots, remember that there is a runaway stabilizer QRH response and pilots should follow that when indicated."

If you're arguing that the pilots didn't know about the possibility of runaway trim, I don't find that very likely. If you're arguing that they didn't specifically know that MCAS could cause runaway trim, I probably agree but think it's irrelevant as the response to runaway trim is not to diagnose or draw the system diagram of the fault, but rather to simply stop the fault, which the stock 737 checklist would have accomplished, if followed.

How much resources would you put towards improving airplanes to save those 400 people?

You would almost certainly see better returns if those same resources were put towards improving cars, or roads, or curing disease, or infant mortality, or healthcare, or education.

In our current and economic system the money that air travel generates is not going to be made available for any of those other fine causes, it's going to either be used to make air travel safer and more environmentally friendly, or it's going to be used to pay dividends to shareholders. It will only be used for safety and environmental standards (and crew care, and passenger comfort) if travellers demand it is and refuse to fly on lines that don't conform.

That actually is a good point. Thank you for the perspective.

Boeing makes safer airplanes than the FAA requires.

For example, twin engines are statistically safer than more engines, and Boeing had to lobby the FAA to allow them over water.

For another, Boeing wings do not have hydraulics that extend past the engines. Torque tubes are used to transmit power further. This is so that a major engine failure won't damage the hydraulic system. This was not a requirement until a crash from another manufacturer's airplane was due to such damage, and Boeing airplanes were not susceptible to that failure mode.

Your theories about how the industry works are speculation.

All theories are speculation, but these are based on observation and contact with the industry. Boeing have acted in exactly the way that my theories predict and dictate; they are certainly reactive to the threat of regulation and customer pressure. If Boeing was to become an "unsafe" brand their shareholders would be very damaged.

The twin engine statistics story that you cite is not the story that I remember for the 777.The FAA had rules which were based on the behaviour of old engines and dictated that an twin engine aircraft should be a smaller distance from an emergency strip than was possible for the best routes across the major oceans (I believe it ruled out the Pacific routes completely). Boeing demonstrated (successfully and correctly) that modern engines failed so rarely that the four vs two distinction was obsolete.

What has happened in recent years is that another failure mode has emerged, it's a complex one due to pilot training, HCI and the vulnerability of automated systems to failure in modes where common sense knowledge could protect them. I expect and believe that the systems and institutions of the aerospace industry will be fully strong enough to manage this, but I don't think that this is the case for many other industries that are beginning to experience the same sort of problems - notably self driving cars and computer medicine.. Many people think differently - the test of theories is time, so we shall see.

In what way do you find the number of air disasters surprising? What number would be unsurprising?

It's not the number, but the number with the same general cause. Any failure of a complex system has multiple causes, and the NTSB/FAA usually responds to incidents with multiple recommendations, to eliminate any possibility of this chain of events happening again. This has been remarkably successful.

It's surprising to me that airline designers continue to add sensors and computers and automation, when in so many of these cases it's the automation which itself is the problem.

Would the NTSB ever issue a recommendation like "Your aircraft is too complex" or "You rely on too many computers"? It seems to me like that could be a valid cause, but it's simply not in their vocabulary.

Overall, flying is safer now than in the days of manual control or more simplistic autopilot. The problem seems to me to be that when things do fail, the cause and effect may not be obvious to the crew so they struggle to understand why the plane is doing what it's doing, and how to correctly respond.

> It's surprising to me that airline designers continue to add sensors and computers and automation, when in so many of these cases it's the automation which itself is the problem.

New features can indeed raise unanticipated problems. But that automation isn't added for the lulz, it is added to address other safety problems. The safety record has gotten dramatically better over the decades, and that speaks for itself.

> Would the NTSB ever issue a recommendation like "Your aircraft is too complex" or "You rely on too many computers"? It seems to me like that could be a valid cause, but it's simply not in their vocabulary.

Neither of those vague hand-wavy recommendations are actionable, so the NTSB would never issue them.

Although vague concerns aren't themselves actionable, the actions that could be recommended in both cases would be research. To look into whether "too many computers" is actually causal and what might be done about that. My guess is "No" but if the investigators thought we need to check that's what they'd recommend.

But most often what's needed is smarter machines, not less of them.

Inadequate engine performance during take off is a periodic cause of problems. Research was done into whether airports could add markers so that pilots could tell if their acceleration was less than expected. That went nowhere because humans don't have an accurate sense of acceleration or time passing. Today new Airbus jets integrate their position over time and will warn audibly "Check acceleration" if underperforming, long before V1 (maximum abort speed). Meanwhile a common underlying cause of inadequate performance is entering incorrect air temperature, modern Boeings will delete input temperatures that don't resemble the measured temperature at an air intake when the engines switch on. Pilots must enter a plausible temperature and try again, or perform a fully manual take off. So investigators now might recommend one or both of these countermeasures be adopted, or that the research be done again knowing these options exist.

The NTSB seems to take the overall view that automation in airliners is generally reliable, though balances that with the view that pilots must be prepared to take over for unanticipated circumstances (United 1549).

https://www.ntsb.gov/news/speeches/CHart/Documents/hart-2018...

Small nit: V1 is the takeoff decision speed, not the maximum abort speed. If the decision is made to abort right at V1, the airplane will continue to accelerate for a second or so while the crew implements the first action to effect the abort/RTO.

Aren't sensors being fitted duplicate/redundant?

Garbage in, garbage out. More things change, more they stay the same.

Do aircraft inertial measurement systems ever provide linear velocity information that isn't trash? I had assumed not (except for special cases where you can do e.g. a zero-velocity update)

An inertial measurement system cannot measure velocity. It's insensitive to which inertial frame it is in. (to move from a non moving frame to a moving frame, you must accelerate, which it could detect. but then you have to integrate these accelerations over time)

Your assertion is simply demonstrated to be untrue.

Inertial systems integrate all measured accelerations over time to compute present velocity and position. They are periodically corrected with external references. The accuracy is limited to the accuracy of the accelerometers, the sampling frequency and the accuracy of the digital math, among other things.

This IMU idea could be used, but I am not aware of any current commercial IMUs in airliners - but I am not an airline pilot so that does not count for much.

If I were given conflicting information I would see how the plane is acting. I have flown planes with various primary instrument failures over my decades, some in actual instrument conditions, and always had sufficient secondary sources of data to determine the situation. As an example in a piston plane: Tachometer fails? Cross check fuel flow, manifold pressure and airspeed - subject to mixture control which can be reasonably set using "lean to roughness and enrich" method. Been there, done that. I have had an altimeter fail insidiously slowly, when water got into the static lines (despite redundant ports and properly routing them upward), while in actual near minimums in a non precision approach. (I used GPS altitude cross reference and odd seeming behavior of altimeter needle to diagnose and switch to alternate static.)

Importantly, if the plane is flying straight and level and sufficiently fast, over time this becomes a "feel" and you can tell if something is extraordinary. Maybe not on airliners, but definitely the piston planes I fly a lot and probably the small jets I fly on occasion.

This is why some of these things are mystifying to me. If your engines are set to a certain power and seem to be operating properly, and your attitude is just so, in the absence of unusual meteorological conditions your plane should be performing in a manner consistent to your last experience. If your stall warning is going off, well, maybe the instrument broke. Air France is really scary mystifying.

Look, I am a software engineer so I get it, people can get locked into a misconception about a fault and not step away long/far enough to consider other alternatives. It happens a lot in debugging. But for whatever reason my brain works much faster and more dynamically when shit happens in the left seat of an airplane having problems.

> But for whatever reason my brain works much faster and more dynamically when shit happens in the left seat of an airplane having problems.

Maybe you didn't intend it but this sounds a lot like "couldn't happen to me!"

LOL. Oops.

I meant to mean "It has most definitely happened to me" although not yet in the context of piloting - but in other contexts.

I already mentioned that acceleration measurements can be integrated to get velocity. I asked a question about the reliability of the velocity information available to an IMU, and your reply did not address that.

Inertial Navigation System (INS) measures linear acceleration (via accelerometers), angular velocity (via gyros) and optionally magnetic heading. Using integration and filtering (Kalman or similar) you get aircraft velocity, position and attitude. Without constant correction (GPS for example) calculations based on INS measurements degrade over time (due to instrumental and calculation errors) but in range of minutes they are pretty accurate.

https://en.wikipedia.org/wiki/Inertial_navigation_system

The comment I was replying to implied that checking with the inertial measurement system was an alternative to checking against GPS.

I still think I'll be surprised if accelerometers provide useful velocity information over timescales useful to a pilot. But maybe that's because I only have experience with cheap MEMS accelerometers from around 15 years ago.

"An inertial navigation system (INS) is a navigation aid that uses a computer, motion sensors (accelerometers), rotation sensors (gyroscopes), and occasionally magnetic sensors (magnetometers) to continuously calculate by dead reckoning the position, the orientation, and the velocity (direction and speed of movement) of a moving object without the need for external references.[1] It is used on vehicles such as ships, aircraft, submarines, guided missiles, and spacecraft."

https://en.wikipedia.org/wiki/Inertial_navigation_system

In any event, what would be the use of inertia-based calculations, when airspeed (through an atmosphere always changing) is the important more bit to keep the plane in the air?

Aviation does not usually require the precision assumed here. The aircraft would have flown fine anywhere between (say) 150k and 300k, so an initial-nav indication of (say) 250k +/- 20k would tell the crew that things are "close-enough".

More useful at the time would be to set a "sensible" power level and maintain approximate altitude ... and looking out the window is usually helpful!

Ground speed (which is what inertial system would indicate) is not very helpful for the pilot. All piloting relies on airspeed which is how fast the aircraft travels in relation to surrounding air.

The difference between ground-speed and airspeed is the wind. In normal conditions, ground-speed is a useful proxy for airspeed - perhaps not good enough for an airliner precision approach, but more than adequate for normal flight.

For the Lion flight, winds were light and the aircraft was at low altitude, so ground-speed would have been very close to airspeed.

In any case (as pointed out by earlier comments) a proficient pilot should have no trouble flying in daylight and good conditions using only the "Mark I eyeball". It appears the aircraft crashed because the crew didn't fly it - a surprisingly common occurrence for automation-dependent crews!

Reply on news.ycombinator.com