New Software Flaw Could Further Delay Boeing’s 737 Max | Hacker News Reader

New Software Flaw Could Further Delay Boeing’s 737 Max

bloomberg.com

81 pointsosivertsson108 comments

This is more and more starting to smell a bit like this is the first time in a while when somebody is actually looking whether a plane fulfills the regulations. Just wonder if FAA & co should actually audit some other models as well with same level of scrutiny? And by all means not just Boeing.

Just American manufacturers as a priority as this is clearly where the revolving door is.

If they'd ever audit (and let alone certify without outsourcing it to the very manufacturer) the 787, it won't be pretty for Boeing.

Being the 787 was a new type rating at the time of its release and its release happened before many of the major cutbacks in the FAA, I’m not sure that’s a fair statement. Do you have a source to back that up?

To back what up? That Boeing self-certified on the 787 too?

The whole “Boeing verifies itself and creates shoddy products” came out in 2013 already. Nothing ever improved since.

https://www.reuters.com/article/us-boeing-hearing/boeing-faa...

How many new planes are designed each year that the FAA needed to abdicate this responsibility?

Aren't planes hard and basically there's only a few new major models in a year?

Two words: regulatory capture. I very much doubt that the career engineers and administrators at the FAA chose to abdicate the responsibility, but there are all sorts of ways that politicians and their appointees at the top can bring this about without an explicit policy: reducing headcount and reassigning staff, rewriting the low-level rules and guidelines, burying the staff in paperwork and meetings... Boeing can also subvert the process by disputing every point, administrative burying, and by concealing things (such as the increase in MCAS' power to change the trim after the initial version was found to be ineffective.)

> basically there's only a few new major models in a year?

Depends on how you define major, but there’s really only a few new major models a decade. Even if you count things like 787-800, 787-900, etc variations as major, there’s still not a few models every year even if you include all major commercial aircraft manufacturers.

There's a steady stream of airworthiness directives from plane manufacturers. Old parts get discontinued, new parts have to be used, issues are found and corrected, more inspection is needed, sometimes less inspection is needed. More training, different training, etc.

> This is more and more starting to smell a bit like this is the first time in a while when somebody is actually looking whether a plane fulfills the regulation

Of course it is. That's the whole point of MCAS, limited pilot training, and the rest of Boeing's "strange" design decisions to make the Max seem like a minor update.

Let me state this explicitly: The 737 does not meet current FAR part 25 standards. It did meet part 25 standards at the time the type certificate was issued 50+ years ago, but the standards have changed.

So how was the Max certified? Well, the FAA awarded Boeing an "amended type certificate" (that's a term of art) for the 737 Max, which means that it is certified on the basis of being a variant of the original 737-100 that flew in 1967.

That means aircraft systems can be grandfathered in, even if they don't meet current standards. Many of the 737's systems would need to be extensively redesigned in order to pass muster for a new type certificate. Boeing did everything they could to avoid a new type certificate for the Max, so as to avoid redesigning the entire plane.

Fun fact: Every Boeing 717 still has a placard near the main front entry door that proudly announces its Douglas DC-9 type certificate (from two corporate mergers ago).

Isn't this (grandfathering tactic) gaming the system? Like borderline fraud?

fraud, maybe not. bad idea, maybe. the idea was that a lot of effort and money was spent certifying a plane, and it makes sense to allow delta changes to the design without revisiting the entire process. so a plane is in service and customers want it to fly a longer distance. the oem makes them a configuration that has more fuel tanks, and less seats. the delta change has to be evaluated against the original certification, because it is possible that meeting current rules is impossible without major rework (think how wiring is done, the kind of hydraulic fittings allowed, things that are everywhere. it allows the economic life of the design and the investment of end users (airlines) to be protected with some semblance of regulation.

That explains manufacturing new product in 2019 with 80286 CPU, got its certificate in the eighties, better not touch it.

Will people even want to board these planes after it became clear how Boeing handled (botched) the development of this plane and how FAA insisted on keeping these planes up in the air (while stating they were perfectly safe as any other) until they absolutely couldn't (i.e. until other countries grounded it)?

I wonder what the game plan is for getting these planes back in the air for carrying passengers. A small vocal minority will make a big stink about it and it will get amplified. Regular folk that don't know or care will join and it will become a huge shitfest when the day comes.

There are also other regulators in Europe and China who will look into this a bit more in detail instead of just rubber-stamping anything that the FAA approved.

I think this is the real issue for Boeing. The FAA will, eventually, certify the 737MAX, but it is not clear to me that European (or other) regulators will feel compelled to follow suit. If the 737MAX can only fly domestic flights, I'm not sure it is economically viable (literally not sure).

No (n=1)

People (not on HN etc.) will happily buy plane tickets on sale that happen to be flights on the 737 MAX once it is certified again.

I still think the general public trusts the certification to be done right and that the plane will only be allowed to fly when experts agree it is safe.

Which is why the stink from pilots and devs with experience from building similar systems is so important.

The bigger issue are business travelers -- who contribute approx. 60% of an airline's profitability through higher ticket prices and more frequent travel. We want to get back to see our families, and we're not very price-sensitive because the company is paying. No way in hell I'm flying on a MAX any time soon.

People may buy discounted plane tickets but most will not trust the certification, in my opinion. I'm avoiding any new boeing planes for my next flight

My view is that when this certification process is over, the MAX will be the only Boeing plane that's received proper review and certification.

What if it never receives certification? What if the way they mounted those big engines on a short plane proves to be a fatal design compromise?

Then they get used for freight and military transport and we all fly Airbus.

Really? Are the regulation requirements for non-passenger-service really that much more lax? A 737 MAX crashing into a city somewhere could easily kill a lot of people in addition to the two pilots onboard.

Dunno about freight, but there are zero certifications for so-called experimental aircraft. You can slap wings on a VW bus and fly it if you want.

You can't sell experimental aircraft though, not commercially anyway. So you wouldn't see 737s being used for freight under this regime.

Oh for sure. But if you’re worried about a random object falling on your head, it’s still a concern.

But of course, practically (/statistically) speaking, you should be much more worried about rapidly-moving objects rolling over the ground.

We're not talking about what I'm worried about, we're talking about what the FAA and other similar airplane licensing bodies will permit, and they are not allowing the MAX to fly for any purpose right now (save the one-time ferry flight to a parking spot until the problems are fixed). These licensing bodies only do airplanes, not cars, and so the safety of cars isn't relevant to them.

But yes, on a personal level, I too find it shocking how >37k people are killed in motor vehicle crashes every year, with countless more injured, and yet we're doing so little about it. There's clearly a lot of low-hanging fruit there.

Experimental aircraft need a special airworthiness certificate. You need an inspection by the FAA, and you have to do a flight test program before being allowed to fly it over populated areas.

I stand corrected, thank you! Unfortunately I can neither delete nor edit my original comment.

I was thinking of ultralight aircrafts, but those have strict weight limits (no VW bus), so the potential damage from one crashing is much reduced.

military planes are certified, but to different requirements. and of course there is a trade off between safety of flight and mission effectiveness. military aviation is inherently a high risk endeavor.

If the Boeing 737 MAX doesn't get certified to fly again I think there's a stockpile of airplanes that could be re-tooled for military operations. Some ideas come in mind

* flying tankers * airborne radars * electronic intelligence platforms * naval patrol platforms

There are some companies trying to use airliners as orbital deployer platforms for small satellites.

I personally doubt the 737 MAX will not be certified but I also doubt that FAA will not require pilot training. That maybe reason for some (but few airlines to cancel their contracts). But you should take my statement with a grain of salt.

Maybe I'm just a very clueless passenger but I admit that I never ever check what kind of plane I'm in before I board it. I have a flight booked two weeks from now and I have no idea what kind of plane it's going to be in. Aren't most people the same? And if so, is the paranoia around the MAX big enough to change people's habits in the long run?

I've never checked this before and I will not be putting my family on this model.

I check every plane. They're VERY different in terms of comfort. Maybe if you fly US domestic it doesn't matter. But for international flights aircraft type makes all the difference. (see e.g. Emirates horrible 777 vs it's very comfy A380)

Where there's an obvious external difference it's easy to differentiate (number of engines in the case you cite). However most domestic US flights are on single aisle twins that all look alike.

You’ll probably notice a difference in comfort (especially noise) between a 737 and a320, say. And you can tell the difference between a 737 and 757 by the way the 757 is unreasonably long (see also, a321).

I check as well. However, the airline can/will change the aircraft at their discretion. Last time I didn't find out until I arrived at the gate. If you don't want to fly on the changed aircraft, I'm not sure what recourse you have at that point. Of course, ultimately you can not board, but there may be penalties.

If you had carry-on only, it might not bother them much. If you checked baggage, they need remove it, causing a delay. Although in principle, this would be "just as if" you missed the boarding call, I suspect if you tell them ahead of time (as in: "I'm not boarding this plane."), they'll try to persuade you there will be penalties. I don't know.

There must be fine-print in the contract of carriage, since the ticket ("product") purchase indicates the aircraft.

Perhaps one could verify the aircraft before check-in. If you don't receive a boarding pass, obviously there is no problem whatsoever, you just lose the flight unless you've paid top-dollar for a refundable ticket.

> However, the airline can/will change the aircraft at their discretion

That’s very rare due to different seat configurations. That normally means a flight was cancelled or something else went fundamentally wrong.

This. I’m willing to pay a few hundred dollars more to fly the A380 on long haul flights when it’s an option.

Domestic flights I don’t really care.

A lot of people (e.g. my wife) do check, though mostly up to now because they have opinions about seating, and you have to check the plane model as part of that process.

I expect the game plan involves a rebranding and a host of minor upgrades. There will still be people pointing out "this is the same plane under a different name", but Boeing will counter by calling those people conspiracy theorists and pointing to the hundred differences. Some experts will side with Boeing on this. Even a sophisticated consumer won't be able to tell reliably which of the "experts" is right.

I bet there's going to be a re-branding of the aircraft to "manage" perception.

They already tried. https://www.theguardian.com/business/2019/jul/15/boeing-737-...

It only works if it flies under the radar. I don't see that happening here. The second they rename the planes there's gonna be lots of articles written about it, and many people are gonna be thinking "the newest model 737s are unsafe", regardless of what that model is actually called.

If they just drop the MAX branding, it'd immediately get lost amongst all the other 737 variants.

With the risk that people take none.

in 2018.

As someone who has watched a lot of episodes of "Air Disasters" (AKA "Mayday", "Air Emergency", or "Air Crash Investigations" in various places) [1], The Weather Channel's "Why Planes Crash" [2], and National Geographic's "Seconds from Disaster" [3] which had a few episodes involving air disasters, getting me on a 737 Max after they release a fix wouldn't require any more force or involve any more kicking and screaming and threats of violence than getting me on any other modern airliner.

[1] https://en.wikipedia.org/wiki/Mayday_(Canadian_TV_series)

[2] https://en.wikipedia.org/wiki/Why_Planes_Crash

[3] https://en.wikipedia.org/wiki/Seconds_from_Disaster

If kayak etc. don't offer the option to filter by plane type now, they will or a competitor will. While plane crashes have happened in the past, I can't remember one that was so clearly gross negligence on the part of the manufacturer. Not everyone will forget that Boeing stopped caring about the lives of their passengers by putting them into the hands of shit software driven by a single point of failure. The simplest and only explanation in this case is malice. No engineering team or organization could possibly be this dumb. I think that alone will make people remember it better than airplane accidents. These crashes were not accidents.

All of Boeing's self-certification rights should be scrapped. Clearly Boeing's internal safety culture isn't in a state where it can be trusted. Mistakes happen & planes fall out of the sky, but this seems like a rather pervasive culture issue.

Privileges, not rights. The rest of us with smaller pockets probably wouldn't be allowed to self certify our aircraft.

Sure we can. If you built an airplane you can self certify it passes applicable regs. It’s a different story if you or I are wrong about our certification because we don’t have the deep pockets to prevent litigation that a multi-billion dollar company has.

You can build a lightweight craft (see FAR 103), self certify as such and fly without ever talking to the FAA. Your lightweight craft could be 100lbs over the limit and no one would know until it crashed and the NTSB got involved. Your lightweight craft could also be a Cessna. You are guaranteed to get into more trouble when the FAA finds out you are grossly violating the law.. especially if loss of life is involved, see Boeing as an example.

No manufacturer should ever be allowed to self-certify.

What kind of teacher lets his kids self-grade?

Well. The FAA in this case is not teaching Boeing how to make planes, so your comparison is not exactly a good one. Maybe an example of another regulated industry like NHTSA or OSHA?

I.e. I don’t see gold mining companies performing their own MSHA audits, but I do see them building controls and processes to satisfy safety regulations

A college or undergrad university doesn’t teach its students how to excel in a GRE or GMAT. And yet you won’t ever get to self grade yourself on those. An external (hopefully somewhat impartial) organisation does.

What kind of educational institution lets its employees assess how well they impart knowledge to their students?

That's why most countries have state exams :)

I do actually remember self-grading in at least a few classes in school, and it was quite effective. You learn more from grading your work (i.e. finding the errors) than if someone else does it for you.

This wasn't for final exams obviously, more like quizzes and such.

Exactly. This is for a final exam. The final certification of a whole aircraft type carrying billions of human beings over its lifetime.

I remember a few times in school when we took turns grading each other's work. That scenario raises all sorts of other potential for problems, but it's an interesting thought experiment: what if Boeing were responsible for certifying Aribus' planes and visa-versa?

Too few players; same collusion situation would occur as Boeing had with FAA.

There seems to be no information on what the problem actually is. This seems a bit odd since Boeing themselves reported it and they should know all the details.

Is it related to the blank screen problem that was discussed a few days ago?

https://news.ycombinator.com/item?id=21991087

Is this reporting with such limited details some kind of damage control along the lines of "we cannot wait longer and must inform the shareholders now" and "we cannot tell the public all the information because it would damage our reputation"?

With the little details we know now we can assume that it is 1) "nothing big" as in an assertion that caught an unexpected new version number and the program simply needs an one-line-patch to update the version check or 2) they detected a problem that turns out to be "unfixable with current hardware" like a weird kind of race condition where the communications can get stuck sometimes but it's hard to reproduce, doesn't happen in the lab and nobody really understands what's going on or 3) "something entirely different".

> The issue involves how software on the plane checks itself to ensure it’s receiving valid data [...] when the system is initially starting up.

> software reviews have occurred in a special simulator used by engineers on the ground.

> The problem came to light when the latest version of the software was loaded onto an actual aircraft

My stomach hurts reading this! I get no confidence at all and I am scared more problems will be found or more accidents occur.

Just guessing... Someone at Boeing testing the new shiny software on a real airplane decided to mess with a sensor (AoA?) to simulate it being stuck or something and to make sure this would be properly handled. It did not work.

How on earth can problems like this be found this late?

I would imagine the simulators will be updated to better reflect the conditions that trigger the error on actual hardware, but it's hard to build and iterate a perfect simulation of complicated stacks like a modern jetliner.

In a system like this the software organization should be super-confident that no matter what failures occur they will handle them gracefully. They should be this confident before even entering the sim.

Getting there is hard and you should be using both models you can prove correct and hands-on "vulnerability" testing. And to make it feasible you should keep your "stack" as simple as possible.

To me your comment reads like developing a system like this in a trial-and-error way until you get a pass from the sim would be OK. It is most definitely not OK.

Of course you don't start from a crude simulation, but when the sim diverges from real hardware behavior, you need to find the reason and update the sim to reflect real world.

the avionics software process has a lot of iteration in it, because of the nature of software. it all starts with a very detailed requirements phase, where what the software is going to do is written down, modeled, and reviewed. design, code, unit test. some iteration here to remove the defects. then it goes into a software test environment that replicates part (or a lot) of the system. There would be a 'software' AOA vane that would be set to replicate different conditions. (running actual A/C hardware for processor and OS) written test procedures and written results from the tests. the it would go on an airplane for specific tests. safe and simple first, becoming more complex.

my experience is every problem seen in the air can be replicated in the on ground development simulator (SIL- Systems Integration Lab). Every one once the conditions are determined.

I think the issue is that they have now changed this risk classification of a number of systems and now have to think of stuff like "well, if the variable tag gets bit flipped, what should happen?"

Its not really a sign that they are finding things that were uncertifiable before, its that they are uncertifiable post-grounding.

I would honestly expect more of this to happen throughout the industry, since the FAA doesn't want to be blamed for anything, they are going to expect lots more to be spent on making even safer aircraft.

"Is it related to the blank screen problem that was discussed a few days ago?"

That's a "NG" specific issue, and not applicable to the MAX.

http://rgl.faa.gov/Regulatory_and_Guidance_Library/rgad.nsf/... (PDF)

This is unfortunate.

I am amazed that along every step of the way, Boeing appears to be shooting itself in the foot.

Having worked in this industry (although not on airplane software), this seems to be a big management and cultural problem with Boeing. It was apparent in other programs and the decisions that the higher ups made.

And the regulator. Is the cultural problem due to American culture? Profit as god.

Which regulator in the US regulates still? Maybe the FDA, but even there I have doubts.

The USDA is a rubber stamping under-funded organisation as has the FAA become. Part of the culture there.

Is it me or is every single issue reported as something exceptional causing boeing to look like the biggest amateurs in the world. What will be next? Boeing migrated all the vital control systems to nodejs. Infiltrated npm packages caused unknown issues. Further delay required.

Happens when you outsource life-critical code to the lowest bidder, who doesn't even work on airplanes.

I wonder what the Gantt charts would look like for redesigning the 737 MAX from scratch with new type certificate.

First revenue flight if everything goes smoothly: 2065? There’s like a 20% chance Everett will be scoured to bedrock by a tsunami by then. So I can understand the reluctance to do the right thing.

Using existing Boeing Standards and Processes? (how to install a rivet. how to anodize aluminum. how to verify software. 100 years of experience written into requirements and specifications of how to build a plane)

Likely 10 years. Starting from scratch. 30 years.

It's incredible they are still pushing to fly this plane without changing the plane itself (engines etc.). This is, as most software issues, an issue of trust. After Boeing killed almost four hundred people with an airplane they knew was unsafe we're supposed to trust the latest software update? Again? That's what they said after the first crash.

This is a prime example of software (these days sometimes called artificial "intelligence" or other nonsense) running amok and the dire consequences it can have.

At what point will Boeing decide to just scrap them all? Are they relying on the flying public being oblivious to what plane they're getting on? I know I wouldn't board a Max regardless.

Not happening. We are talking about many billions of dollars of capital investment, even if we only count already built aircraft.

When a major regulator (probably not the FAA, maybe European) finally announces, "we are never certifying this plane model".

No way government allows it to be scrapped. Worst case scenario they will be refitted for cargo duty, or 'purchased' by army/navy at a reverse discount to prop Boeing up.

> or 'purchased' by army/navy at a reverse discount to prop Boeing up.

What use would the military even have for such a plane? The only thing I can think of is ferrying personnel out to deployments, but I understand 737s are fairly short range planes and better, cheaper charter flights are probably available.

The US military has a hundred or so 737 derivatives:

https://en.wikipedia.org/wiki/Boeing_737#Other_variants

Open source it. Just open source it all. This would restore so much faith and allow passionate researchers to improve safety.

I will never fly on one of these air Yugos. I suspect there are quite a few like me.

I am guessing that previously they did not take the testing nearly as seriously.

For the sake of all of us I hope this plane never sees another flight.

I'll never set foot on a MAX series out of principle. Not much else one can do as a consumer other than vote with one's wallet.

Why are they even bothering with the MAX?

Do they really think that people will willingly board a MAX plane again when travelling? I'm certainly not going to do that.

Because they have hundreds of new airplanes sitting on the ground, as well as a tooled up production line as well as the innumerable companies involved in the supply chain.

The 737 Max issues have been causing layoffs at supplier companies that were heavily reliant on the Max as a source of income.

Yeah but nobody is going to get on that thing. The moment someone tries to use that model again there will be headlines like "XX Airlines puts death plane back in the sky" etc.

It's a waste of money trying to get it back on track.

> Yeah but nobody is going to get on that thing

People absolutely will. Boeing is literally too big to fail and people don't care enough either. If someone can have a 50 Euro flight from Berlin to London they won't mind if it's a reintroduced 737 MAX.

We'll see about that. Why would airlines accept to operate these planes at a significant loss after the loss they incurred with this whole fiasco to begin with?

I personally wouldn't board this plane until it had a spotless record a decade in the air even if they paid me for it. And I'm someone who loves flying so no specific phobia involved.

> Why would airlines accept to operate these planes

Because their other option is getting at the end of the line for an A320neo and get delivery in 6 years (or worse)

A lot of people forget that Boeing doesn't sell 50 Euro flights. Or any flights, or any 50 Euro item.

The people who buy carelessly for 50 Euro aren't Boeing's customers.

The people who are Boeing's customers are airlines, who pay eight- and nine-digit sums and perform due diligence. "Well, one group of customers won't care" is not due diligence. So if, say, Lufthansa concludes that the 50-euro customers won't care but the 500- and 2000-euro ones might, that has an impact on what price Lufthansa is willing to pay.

Ryanair, the largest supplier of such flights, has never had a fatal accident, and is quite proud about that. They’re sticking by Boeing for now, but if the ultimate fix is questionable, who knows.

They had however incidents involving underfueled machines.

And there's the problem for Boeing: They better make sure that no more MAX's fall out of the sky due to their fault otherwise airlines will skip the machines. What would happen with Ryan Air if they had one or two accidents like those happened? Knowingly using machines with the past of the MAX. Rage would be allo over the net, I am sure. And Ryan Air would be in a very tough position.

Yes but that only explains the magnitude of this disaster for Boeing. People will not merrily board these planes just because Boeing is between a rock and a hard place, because of their own wrongdoing no less.

What fraction of airline customers even try to find out what airplane model their flight will be on? My intuition is it's irrelevantly low...

They'll just rebrand it, an average flyer won't know the difference between 737MAX and the older model by looks. At best they might be able to tell that it's a 737.

Cheating is what has gotten Boeing into this place to begin with. Another cheat (sorry, "rebranding") won't exactly increase confidence.

Oh don't worry flying public. This isn't the shit 737max. This is the 737-8! Totally different thing! Believe me, it's true.

Boarded a plane last night. Looked like a 737. Breathed a sigh of relief when I saw it was an Embraer.

There are multiple styles of 737, most without the systems or problems from which the 737-Max 8 suffer.

How will Boeing and the regulatory bodies handle problems their extensive review finds with the MAX that may also exist on the NG?

Hopefully well? However, the what-if game is generally bad to play, especially when we can't even ball-park frequency. There are always many what-ifs that can be made.

However, even when we know frequencies, we often still ignore them. What if an accident happens on the way to the airport, which is far more likely, and far more likely to be fatal, than flying on even a 737-MAX 8?

What I think is more probable than any issue with the MAXs translating to the NGs is: what-if other (recent) FAA certified planes, by any manufacturer, have undisclosed problems or systems? There's obviously a systematic flaw in how the FAA conducts certifications, so this worries me more than finding a fatal and fundamental flaw on aircraft that have been in near constant use for 20-25 years old at this point, as are many/most of the the NGs have.

The rudder hard-over in '90s 737 should be fixed now, no problem :> https://en.wikipedia.org/wiki/Boeing_737_rudder_issues

I'd call that one even more scandalous but the media didn't seem to report it much at the time.

Airplanes were much more dangerous back then, and those 3 accidents happened over a much larger number of flights (orders of magnitude larger) than the 2 Max ones.

How does an Embraer look the same as a 737? Very different profile inside and out. The seat config alone should have been very obvious.

Reply on news.ycombinator.com