Boeing's software fix for the 737 Max overwhelms the plane's computer
moonofalabama.org
moonofalabama.org
-----------
Boeing says that it can again fix the software to avoid the problem the FAA just found. It is doubtful that this will be possible. The software load is already right at the border, if not above the physical capabilities of the current flight control computers. The optimization potential of the software is likely minimal.
MCAS was a band aid. Due to the new engine position the 737 MAX version had changed its behavior compared to the older 737 types even though it still used the older types' certification. MCAS was supposed to correct that. The software fix for MCAS is another band aid on top of it. The fix for the software fix that Boeing now promises to solve the problem the FAA pilot found, is the third band aid over the same wound. It is doubtful that it will stop the bleeding.
The flight control computers the 737 MAX and NG use were developed in the early to mid 1990s. There are no off-the-shelf solutions for higher performance.
Boeing's latest announced time frame for bringing the grounded 737 MAX planes back into the air is "mid December". In view of this new problem one is inclined to ask "which year?"
-----------
Ouch...
Introducing a new CPU into airplane like 737 will require another whole series of test/certifications.
And right about now, Boeing management is thinking we could use some of those senior engineers we laid off because we thought they were not needed as our products are now mature.
It’s not like the wrote any of the doomed software or anything.
They desperately need their multi-million dollar bonuses so that they can afford to continue to live such lavish lifestyles.
What’s wrong with you people? You act like a few lost lives have some value or something.....
/s
But I have problems accepting this fact for civilian planes, especially if the manufacturer just wanted to save development costs. Because you could simply design a more stable plane. Not trivial, but also not impossible.
Another commenter actually said that it isn't really true that the plane is inherently unstable. If so, the error was probably with regulations about classification and Boeing wanting to take a shortcut.
The fact is, every single airplane ever built is unstable in the wrong circumstances. They stall, they spin, they lose lift, they suffer mechanical failures. Aeronautical engineering, rigorous maintenance, and thorough pilot training have resulted in the safest transportation system that has ever existed, but the only completely stable aircraft is one that's sitting in a hanger.
[0] https://en.wikipedia.org/wiki/T-tail
[1] https://aviation.stackexchange.com/questions/1400/how-do-con...
[2] https://aviation.stackexchange.com/questions/35764/are-helic...
[3] https://en.wikipedia.org/wiki/Coffin_corner_(aerodynamics)
[4] https://www.faa.gov/regulations_policies/advisory_circulars/...
Note: this is not necessarily a stupid decision.
If you think they are at best 40 MHz 286 parts, the best that was available by then, running software derived from previous releases that ran on slower chips, there is plenty of room for improvement.
IIRC, you can get a modern 65816 (the 16-bit descendant of the 6502) in 300+ MHz. And I wouldn't expect it to be on a cutting edge process.
That considered, it's entirely possible most of the time is being consumed by reading the sensors. In that case, the sensor communication would need to be upgraded and recertified.
Fun thing: I remember a 40 MHz 286 desktop at one point. Wikipedia tells me they only went to 25 MHz. That's some overclocking...
Fun fact #2: with a feature size of 1.5 µm, one could fit an entire 6502 done in 7 nm process on top of a single 80286 transistor.
(One thing that springs to mind was that the way addresses running past the end of a segment worked was different)
A few executives might shave a fraction of a penny off their multi-million dollar annual bonuses. And a lot of lower level workers are going to lose their jobs.
But no more lives will be lost if Boeing fails to consolidate their defecation on this matter.
The fact that they neglected to do this... I don't know, public perception and Boeing's own representation seemed to paint a picture of "computer" and "chip"-aversion up to the point when they no longer could ignore the issue by designing a plane that needed to have it in order to be certified they way they wanted it to. And by then it was of course already way too late to certify another chip...
Then again, on the other hand: Boeing designs a lot of military planes, too, aren't these almost always exclusively fly-by-wire? Shouldn't they have the know-how in these things as well? Or is there a no information-exchange policy in effect between their military and civilian teams?
The issue is the extensive costs of changing from one CPU type to another. Certification for these systems is a multi-year process and can cost millions of dollars even before any kind of success is guaranteed.
There are still very old CPU's out there, functioning just fine for decades. Safety programming doesn't just swap out parts like consumer computing does - it takes a lot of work to change CPU's.
I don't think Boeing has been kicking the can down the road on the upgrade. I do think they've been trying to cut costs and exploit their customers by offering extra safety features as upgrades, rather than making them standard. Its interesting to note that some of their customers don't require such stringent safety features in the regions they operate - i.e. this is as much of a legislative issue as anything else. It could very well be that Ethiopia doesn't have the same safety requirements encoded in its laws governing flight as France does, so Boeing offers different features not just according to budget but also legislation - although we are sure to see that change rapidly now.
“It Is Difficult to Get a Man to Understand Something When His Salary Depends Upon His Not Understanding It.”
There's issues throughout practically every step and they want people to believe that they're just going to push a software update and everything will be 100% fixed.
Myth: the 737 MAX 8 is not inherently stable, has relaxed stability, etc. Fact: it's very much inherently stable.
Myth: the 737 MAX 8 is easier to stall than other planes. Fact: no it isn't.
The pitch-up characteristic of the MAX 8 is less strong than of e.g. the 757 and that plane flies just fine.
The actual problem with the MAX 8 is that Boeing added MCAS to allow it to share a type rating with the rest of the 737 family (allowing existing 737 pilots to fly the MAX 8 without additional training), and they fucked up MCAS. There's a number of solutions on the table, including removing rather than fixing MCAS and giving up the 737 type rating.
I am continuously astounded that even on HN people are focusing on news cycle bullshit about inherent instability instead of the actual issues with Boeing/FAA that caused this situation.
Evidence so far suggests that MCAS was originally a non-critical system that was found to be too weak during flight testing, and given significantly more pitch authority. For whatever reasons, this didn't trigger the reclassification of MCAS as a critical system and it all went downhill from there.
Here's a pair of sources slightly more credible than the bullshit news cycle:
How is it surprising that people are sceptical of claimed native flight characteristics after the way Boeing has handled communications after the crashes? I don't think that there are any independent facts around, even airline pilots who had been flying MAX from day one would not have much (if any) experience at the edges of the flight envelope. One data point we do know is that some people at Boeing were apparently very concerned about MCAS ever being off.
Boeing needed to counter the A320neo and they needed to take shortcuts that ended up killing people. I wouldn't be surprised if this were management failure and the actual engineers at Boeing have always been throwing around copious WTFs when building the 737 MAX 8.
"Hey engineers, improve the 737 to be X% more fuel efficient and make sure it doesn't need to be reclassified as new type. Have fun!"
No. They chose to take shortcuts.
"The 737 MAX was a bit too easy to pull into a stall when flying with high AoA and making abrupt maneuvers. The larger engines for the MAX hung further forward from the wing, added a destabilizing aerodynamic area ahead of the center of gravity, destabilizing the pitch moment curve at high AoA.
Boeing and the certification authority, FAA, decided added margins was called for. Boeing added a pitch augmentation at high AoA called Maneuvering Characteristics Augmentation System, MCAS.
The aircraft should trim nose down to increase the stick force needed once it passed into the light grey area where the base aircraft had a region of less stability. Before the augmentation, the pilot felt if the aircraft wanted to fly into the stall, it got easier to increase the AoA after 12°AoA. With the augmentation the felt extra force was the same for the first and last part of the curve before the maximum lift was achieved at stall (and stall warning kicked in)." [0]
The manner of the fix (MCAS transparently pushing the nose down) was designed to avoid pilot retraining and thus keep the same type rating.
Edit: The fact that the 737-Max needs a handling tweak is not a failure. Modern planes have all kinds of these tweaks, whether aerodynamic (such as strakes), mechanical (stick shakers) or enabled in software. As the cited article continues: "So far so good. It's common an aircraft’s flight control system has fixes to stability margin changes in different parts of the flight envelope." The problem is that Boeing had a pretty severe collapse of its systems engineering regime.
"The implementation for the 737 MAX had two problems, however:
- The fault checking of the triggering AoA signal was not rigorous enough. This problem has been discussed a lot. No need to add anything.
- The judgment the pilots would identify a problem with the augmentation as a trim runaway and shut the trim off was wrong. Why the pilots didn’t see MCAS rouge actions as a trim runaway is poorly understood."
(The article was published in February. Since then lots of information has come to light about how MCAS determinedly fought correction, and the huge mental and physical loads imposed on the pilots.)
Edit 2: FAA regulation mandating increasing elevator forces for all transport aircraft: FAR §25.253 High-speed characteristics, (a) Speed increase and recovery characteristics, (3):
With the airplane trimmed at any speed up to VMO/MMO [maximum operating airspeed], there must be no reversal of the response to control input about any axis at any speed up to VDF/MDF [maximum airspeed demonstrated in testing]. Any tendency to pitch, roll, or yaw must be mild and readily controllable, using normal piloting techniques. When the airplane is trimmed at VMO/MMO, the slope of the elevator control force versus speed curve need not be stable at speeds greater than VFC/MFC [maximum control airspeed], but there must be a push force at all speeds up to VDF/MDF and there must be no sudden or excessive reduction of elevator control force as VDF/MDF is reached. [1]
[0] https://leehamnews.com/2019/02/08/bjorns-corner-pitch-stabil...
[1] https://www.ecfr.gov/cgi-bin/text-idx?node=14:1.0.1.3.11#se1...
Your reference suggests that this tweak is to match the expectations of pilots certified to fly 737s, which handle in a certain way.
What am I missing?
I am not an aviation expert in any sense.
§25.175 Demonstration of static longitudinal stability.
Static longitudinal stability must be shown as follows:
(a) Climb. The stick force curve must have a stable slope at speeds between 85 and 115 percent of the speed at which the airplane—
(1) Is trimmed, with—
(i) Wing flaps retracted;
(ii) Landing gear retracted;
(iii) Maximum takeoff weight; and
(iv) 75 percent of maximum continuous power for reciprocating engines or the maximum power or thrust selected by the applicant as an operating limitation for use during climb for turbine engines; and
(2) Is trimmed at the speed for best rate-of-climb except that the speed need not be less than 1.3 VSR
These are transports, not fighters. The basic idea is that you want it to be harder work to make a sharper maneuver, and want the aircraft to naturally level out. If the force required decreases for a steeper pitch/roll/yaw, then the plane will naturally want to intensify the maneuver. It's like a car with oversteer--let go of the steering wheel and it will make a sharper turn. As the article cited said, the FAA and Boeing's test pilots weren't happy with the yoke forces in certain situations.
Now is it unusual for sensors and computer controls to help meet the airworthiness requirements?
I’m guessing it’s definitely not preferable.
I also think everyone is agreed that Boeing (the company and decision makers) really fucked up here, but it seems like a chain of bad decisions at every stage has played a part in this disaster.
[0] https://en.wikipedia.org/wiki/Stick_shaker
[1] https://aviation.stackexchange.com/questions/6391/what-is-th...
[2] https://en.wikipedia.org/wiki/Lockheed_F-117_Nighthawk#cite_...
Cite this. Specifically this. The rest of your comment agrees with mine without this being true, and I have not seen any evidence of this being true.
[0] https://www.nytimes.com/2019/06/01/business/boeing-737-max-c...
The 757 ( and 737NG etc ) has constant pitch rate. The Max does not, which is the root of the problem.
The actual forces involved aren't pertinent.
Planes don't need insane multitasking processing power like our smartphones or PCs. They mostly do signal processing and sensor fusion in a tight loop which is quite trivial even for legacy CPUs as it's basic flight math equations which results in highly optimized code.
In terms of aero chips, basic is always better as you want a silicon that's tried and tested for decades to have a deep understanding of it's quirks and bugs so you know the code execution is reliable.
Manpower could also be a bottleneck. Design talent is not what it used to be in the first world after much of production moved overseas.
Yes. Legacy CPU's have the advantage of years and years of testing. Newer CPU's are not as reliable for safety-critical systems inasmuch as not all the kinks have been worked out, and there may be catastrophic bugs in newer designs that won't be discovered until mass deployment on consumer markets.
Meltdown and Spectre have sent shock-waves through the safety-computing industry. I wouldn't want to fly on any plane that is running on the latest-generation Intel chips - they're just not settled yet.