Latest 737 Max Fault That Alarmed Test Pilots Rooted in Software

My jaw literally dropped. It actually took me a moment to realize I was sitting gape-mouthed reading that. We're talking airplanes here, not some LAMP stack.

If this is what we can expect going forward, the Jet Age isn't going to last another 50 years. This kind of software ought to be developed using a tool like TLA+.

That describes the whole 50+ years old 737 to it’s core.

It's proper. This is exactly how you do things with life-critical embedded software. You detect when things are crazy, which of course shouldn't happen but nevertheless does, and you react in a way that is hopefully not disastrous.

The alternative is to ignore the problem.

Sensors go bad. Actuators go bad. Voting hardware goes bad. Message routing goes bad. Even the RAM sometimes goes bad.

When the hardware is failing ("specific streams of erroneous flight data") there isn't going to be a reliable solution. You can't even fully enumerate all the possible failures. How could you possibly guess that a message routing chip now flips bit 7 in every angle-of-attack measurement?

Boeing's answer is standard: when the values look crazy, stop doing stuff that might make the situation worse.

Wasn’t aware the compute power was low on the 737. Interested if you have some more info?

Always thought they would spec a substantially more processing capability than required. I understand economics are a factor but surely it’s not through necessity to save power?

Yeah, it’s a nice way of saying “this is a hack”.

I wonder if the human life calculation worked well this time. As far as I see, Boeing lost more than the sum of the human lives; they also lost reputation for everything new they’ve designed in the last 7 years being corrupted, and they also engulfed the reputation of FAA with them, whose agents would fit the definition of “corrupted” by any people’s definition (I know, they are not, they just used agents of Boeing to inspect Boeing because they were understaffed), and the FAA showed the last step of failure by not admitting that the plane had to be stopped until a few days after the European agencies.

In other words, even in financial terms, it cost more than damages. It may have cost the entire company. They “DeHavailland”’ed their company. Ever heard of DeHavailland? No? That’s probably to do with their 4 successive deintegrating planes that “CEOs have complete trust in.” It just died, as a name. The risk is high.

Do you work at Boeing? ;}

To be fair, any hardware running a WAF in Prod these days is going to have more compute power than a 737.

I see this sentiment a lot, but its kind of surprising. Traditional wisdom states the opposite - never do a full rewrite, you lose the years of buit-up fixes and edge cases and have to learn from scratch in too many places.

Despite everything, I still don't think the solution to the 737 MAX problem is to start over. The base problem that Boeing has its actually their terrible engineering culture, the one that values shortcuts and profit over thoroughness and safety.

Even if they started from a blank piece of paper, they'd have that engineering culture problem (and to be fair, it is far more likely an engineering management culture problem), and they'd still turn out a design that kills people.

Can you forward the data Airbus, your car or public transport vehicle manufacturer, Otis or equivalent elevator company obviously sent you for their respective operating environments?

The only reason Boeing is in the “direct control” business is because their best seller is from the 1960s and they don’t dare touch it. Their newer wide bodies 777 and 787 are fly-by-wire and in the latter case even fully electric to a fault.

Agreed. However, the worst part is the FAA. Specifically, what baffles me is the massive hand-wave they did on this considering it is one of their mandates to check flight-worthiness of Airplanes/Airlines.

I hope all the other agencies worldwide will double check all the prior certifications issued by the FAA, because all their credibility is gone. Especially considering the US was the one the last countries to ground the 737 MAX.

I wonder what happened there. Bribes? Favors? Lobbying and ties? All of the above?

That's exactly what they are. How many planes have malfunctioned in the same way and haven't nose-dived into the ground? So far, 0 planes that have had this issue haven't been lost (first plane had the issue, was recovered using non-standard procedures, then same plane killed everyone, even after sensor was replaced!)

You appear to be making a straw man argument that the planes are safe by claiming to refute the 'sudden-unavoidable-death machines' argument which no one is making.

The probability of a total loss of an aircraft is basically proportional to either its flight hours or number of takeoffs and landings (which are (handwave) correlated (unless you are Aloha Airlines)). A fleet of thousands of planes (especially a 'mature' design), flying thousands of hours per year, in service for decades should have very few losses due to fundamental design issues. It would be nice to say it should be <1, but the reality is the stars sometimes align and several weaknesses conspire to make a fatal event. With MCAS, the 737 Max 8 was already at a rate of 2 events in less than a year with a fleet that was a tiny fraction (<6%) of its ultimate fleet size.

Assuming nothing else changed, at full fleet size, that would be over 30 losses a year, or well over 1000 losses over the fleet's lifetime. They misengineered the system by at least 3 orders of magnitude. This is, at the very least, profound professional and managerial negligence. I'm not not yet convinced it's not also criminal. At the very least, all management who touched this subsystem or were part of the no-new-training-at-any-cost push should have already been shown the door up to and including the CEO.

And yes, things _are_ being done and they are adding 'nines' to the reliability as quickly as they can, but again, they are starting from a subsystem that kills everybody on board every 800,000 flight hours or so.

They appear to be dishonest with the public (and perhaps themselves) about how much risk is eliminated with each change and, similarly, appear to not be factoring in that some of these changes introduce other risks and failure modes.

What we have here is a giant fucking mess and a management organization that fosters no confidence in its ability to navigate itself out of it. In the end, I do expect Boeing to somehow glue-gun 3 more 9's of reliability onto this airframe and stop bleeding valuation, but it won't be pretty and it probably won't be fast.

To be fair, if another 150 people die that will be the end of the 737 program - and probably Boeing. 300 more deaths is highly unlikely.

The plane has no fundamental mechanical flaw, it has a software bug.

The only flaw the plane had was being treated as 'just another 737'

Heard of a private GitHub?

Most probably wrong. Code and requirement tracability is mandatory when we do safety critical system. If you have an audit, they can pick a source file or even line of code and ask you to find why it was added and the associated requirement. They can also ask you to show the test report where it was tested