Delays in Boeing Max Return Began with Near-Crash in Simulator
bloomberg.com
bloomberg.com
> In one scenario, the plane aggressively dove in a way that mimicked what happened in the crashes on the grounded jetliner, the people said. While such a failure had never occurred in the 737’s history, it was at least theoretically possible.
> Because at least one of the pilots who flew the scenario in a simulator found it difficult to respond in time to maintain control of the plane, it needed to be fixed, according to two people familiar with the results.
However, the next sentence seems to belie the implication that this was just an edge case:
> The answer was to modernize what was a relatively antiquated design on the 737.
So, of all the things that caused Boeing/FAA to realize and the accept the 737 needed to modernize its decades-old design, it was a simulated edge case involving "gamma rays from space"? Hate to be overly cynical, but that sounds like the kind of CYA thing you'd say when you don't want to admit it you made a huge mistake by deferring modernization in the first place, of which the MCAS concept was a major and disastrous symptom.
Soft-errors are a prime concern when designing safety-critical systems, both in hardware and in software. It's not some outlandish edge case.
Incidentally, while soft-errors in an airplane are probably primarily particles from space, on the ground we're still sometimes talking about it, but in reality it's mostly chip packages that are low radioactive and emitting a alpha particle or so.
But it's not the gamma rays that are the reason for the systems redesign, it's the fact that "at least one of the pilots who flew the scenario in a simulator found it difficult to respond in time to maintain control of the plane".
Boeing's assumption that pilots could respond quickly enough to a loss of control in the cockpit was an underlying motivation for designing MCAS with inadequate safety measures.
That's why they have to rearchitect the system. If those bit-flips happened to the Flight Computer in command, they could lose the plane. Therefore, redundancy must be built in.
It wasn't originally evaluated as posing a catastrophic outcome in the safety analysis, which is why no one paid it any mind.
And I'm guessing their not using pilots of average skill here...
[0]: https://www.reuters.com/article/us-ethiopia-airplane-faa-boe...
...but on mission critical testing with billion dollar implications. You want to tell me they're pulling that name out of a hat?
People have nearly died because of exactly this situation. See, for example, Quantas Flight 72 [1], in which a pair of sudden, uncommanded pitch-down manoeuvres that caused severe injuries -- including fractures, lacerations and spinal injuries -- to several of the passengers and crew.
A long and detailed investigation essentially concluded that these events were most likely caused by the CPUs not being sufficiently rad-hardened. The pilot was diagnosed with PTSD, caused in part by being totally out of control, and quit his career [2, 3].
[1] https://en.wikipedia.org/wiki/Qantas_Flight_72 [2] https://www.smh.com.au/lifestyle/the-untold-story-of-qf72-wh... [3] https://www.harpercollins.com.au/9780733339745/no-mans-land-...
"A number of potential trigger types were investigated, including software bugs, software corruption, hardware faults, electromagnetic interference and the secondary high energy particles generated by cosmic rays. Although a definitive conclusion could not be reached, there was sufficient information from multiple sources to conclude that most of the potential triggers were very unlikely to have been involved. A much more likely scenario was that a marginal hardware weakness of some form made the units susceptible to the effects of some type of environmental factor, which triggered the failure mode."
Cosmic rays would be a very convenient excuse. (Cue the ECC RAM lobby, which will probably downvote this.)
If not, and thus the affected computer system has been added in the MAX, why not design it properly with 3+ redundant copies of the system given that it was designed in the 2010s when both engineering theory and technological resources are plentiful?
If it's seriously the main concern, wouldn't it be much easier to just encase the computer in several inches of lead shielding?
If TFA is to believed, Boeing has embarked on a project to take what is currently a master-slave failover design and jury-rig it into a master-master real-time system. This is based on bad performance in the simulator—not because of MCAS—but while testing errant memory errors from a simulated gamma ray strike!
The obvious question of how can dual-master ever work when they disagree if there isn’t a 3rd source of truth to vote out the failure? This means the software running in both computers has to somehow agree which one is wrong, while one of them is in a potential failure state. This... isn’t how flight control systems are designed from first principals.
I’m not a flight system designer, but there must be hundreds of physical, electrical, and architectural considerations taken at every point in the design process which enable multi-master controllers with the ability to vote out a failure. Everything from the particular sensor suite, the number of sensors, the way they are wired, the way data is acquired and bused through the system, the timing and synchronization of the system clocks, the way that control outputs are calculated, queued, and ultimately issued to downstream controllers... none of the necessary pieces will be in place in a system which up until now makes you select a single master controller before embarking on your flight.
The only conclusion I can reach is that Boeing has lost its mind, and this project is absolutely doomed to fail. The architecture that Boeing has apparently committed itself to now is extremely difficult to design from first principals and a blank slate. I just don’t see how it’s something that can realistically be papered onto a legacy dual-computer system after the fact.
The part in TFA about “adding a wire” practically made me spit out my coffee. The sheer level of arrogance that Boeing management must have to think this would be possible - is just a classic example of an elitist MBA management group totally disconnected from any technical domain expertise. Boeing said they would have this ready for certification by the end of this year?!
The benchmarks and excitement around AMDs offerings are a great example. When you see something that presents a paradigm shift, often the market will not have fully digested it.
AMD is one of those cases, and I believe BA is as well. Their stock is essentially unchanged from where it was before the Lion Air crash in October 2018. This company is worth $200B and we’re talking about 5,000 units of a $100mm plane (that’s $500B in revenue right? I’m not adding a zero there?)
My layman understanding is that actual prices would be around 40% off list price for models that sell really well, with little competition (for instance A321). Much steeper discount above 60% for models that have a hard time selling (A330NEO) or for end of line to close a production gap (777-300ER is very cheap to buy while Boeing is working on the 777-9).
https://www.aviationtoday.com/2019/05/06/boeing-angle-of-att...
That's not really a thing in modern airliners. The computer is still heavily involved even when hand-flying the plane.
As for "modern", the A320 had control laws in 1984 and the 777 in 1994.
Computer Scientists aren't usually taught engineering ethics and definitely get fired all the time for arbitrary reasons.
OTOH, I'm skeptical those courses change behavior, and in this particular case others in comments are saying this is a common architecture in the space, so how could anyone solve this ethical conundrum if getting to the base level of whether this is an ethical conundrum to begin with is difficult?
In that environment, serious ethical concerns could easily be brushed aside as "coders just being coders". When Google got found out for making drones, the employees of a company staged a 5 minute walkout.... that was it, I'm sure some people quit, but not enough to make any impact
>marketing themselves as "COOL TO WORK FOR"/"WE HAVE BEER ON FRIDAYS",
"the idea that you are small and need management to tell you exactly what to do everyday, because you're a dumb idiot who spends all day watching anime and playing video games" does not follow.
Its also my understanding that most do.
BTW: SE typically means Software Engineering in this context.
Note that the UofI has always been in the top ten nationally for CS; its 2020 rank is #6 (behind MIT, CMU, UCBerkeley, Georgia Inst. of Tech, and Stanford).
1: 2005: https://web.archive.org/web/20050321215749/http://courses.ui...
2: 2019: https://cs.illinois.edu/academics/undergraduate/degree-progr...
I liked my ethics course in the philosophy department, but that's more about J.S. Mill, and Kant and the rest. Engineering ethics was a course any Eng. degree had to take, interesting, but hard to apply also. You go through case studies and look at them, but to have any productive class discussion they kinda have to be broken down to be at least somewhat clear and concise.
I don't feel like getting into here, but I feel like even this Boeing issue, which is pretty clear, is less clear than any case study in a textbook.
And, 100 scenarios instead of 10? I sure as heck there's more than that on an architecture redesign!
The National Enquirer may be more reliable than Bloomberg at this point.
>Collins reorganized business units and retrained managers, with an eye toward efficiency and speed. It pushed its suppliers to do the same, and established partnerships with companies such as HCL Technologies, which provides outsourced, lower-cost engineering services from India.
>Along the way, Collins unseated Honeywell as the provider of flight control computers on a predecessor of the Max, the 737 NG, and supplied numerous systems for the Boeing 787, which went into operation in 2011.
>When Collins secured the contract for the Max displays in 2012, it credited the belt-tightening. “There were a lot of cost-saving measures — a lot of tough decisions — that had to be made,” one manager said at the time in a company publication.
https://www.nytimes.com/2019/11/07/business/boeing-737-max-c...
If there was a problem with the master-slave operation on the MAX, not necessarily connected with MCAS, might not the same problem exist on the 737 NGs?
the MAX added new engines, which required forward placement on the wings, which changed COG, which necessitated MCAS.
It's not my favorite solution for every application, but it's a good tool for a handful of situations. And it's a good way to avoid having to think about picking who to trust with only two sockets. There are other approaches, of course. For example, if your system can afford more than a couple clock cycles of latency (e.g., if you don't believe you can crash a plane with a couple milliseconds of control error) you can do the same thing with well-designed software and a bit of extra hardware. It gives you more freedom since there's more hardware to buy, at the expense of engineering and test effort to prove that your solution does what you want.
Doing this over large physical distances also adds some challenges. There are debates around dedicated signal wires direct between control centers vs data networks. Every situation is a little different, so there aren't really easy answers that you can swoop in and prescribe as best-practices. In general I find that people are more-skeptical of packet-switched solutions (e.g., Ethernet) than I would be. But most folks are also better at reasoning about the failure modes there than they are at thinking about the lower-level issues that Ethernet solves for you. And of course there are tons of different signaling systems - not just Ethernet - to consider.
Consider just a few different possible applications and the constraints that they impose. A car has less severe failure modes (many crashes are nonfatal) but less freedom in time. Your trajectory is simply more cluttered. A passenger plane has dramatically more energy, so failures are typically more severe, but you have a lot more time to deal with them because your trajectory is empty (much of the time, anyway). And if you're sane, you've designed it to be passively aerodynamically stable because twitchy maneuverability isn't critical when you're not a fighter jet. A rocket is a different animal altogether. Those are often rigged with self-destruct explosives, and you don't really get much time to change your mind if you erroneously choose to trigger them. Plus they're typically less stable (performance is more dear, gravity isn't helping you, and purely aerodynamic solutions don't take you outside the atmosphere) and are thus more dependent on their control systems.
If the majority of computer aids turn off, the plane doesn't simply drop out of the sky. It reverts back to a more complicated, but human-controllable, vehicle.
From the article (and who knows how much Bloomberg understood) the active-active sounds more like recognizing, not recovering from, errors.
Which isn't the worst idea. Pilots being, well, pilots.
So you're asking the pilot to, in the midst of some other failure, to start flying a weird plane they ever flown before. You could solve that problem by training all your pilots in this BUT this solution is very expensive, so expensive the airlines might just demand their money back instead. And that wouldn't be a solution for Boeing.
And now, the FAA (or EASA) might just mandate that anyway.
I wouldn't be surprised if CAAC refuses to ever recertify the 737 MAX, especially if a trade deal isn't struck.
People forget a project was late if it succeeds, they never forget a project on time that failed because it was rushed, buggy and not polished production.
MBA/business and marketing need to go back to letting the product people and engineers create value, then they can extract the value and sell it. It is much easier to sell value of a good product, it basically sells itself. For some reason American business has completely forgotten the rule this millennia, product then marketing and selling.
You see this same thing happen at game companies when they put a date out there and it is driven by management and marketing over reality of engineering/product/design that makes a fun and solid game.
Boeing was greatly diminished in engineering/product excellence by the McDonnell Douglas management control takeover, everything newly produced since has the tell of this.
#notallMBAs, but the surest way for your product to fail on the market is for your company to go bankrupt before you’ve polished it enough.
In this specific case it was about greed, but sometimes MBAs push for release, because according to the engineering teams the product is never truly perfect
Product people and engineers know how to manage time probably better than business because they are doing the work.
If something fails because it doesn't meet the market there are numerous reasons for that besides not hitting the date.
A good product, that hits the date, can still fail due to market timing or market movement and many other things. Whether a project is launched at a certain date has little bearing on the success of that project long term, short term it might meet marketing and financial quarterly goals, but long term success is always better when it is a good product, and for it to be a good product the people that make products and create value should determine or at least have some say when it is complete and ready.
The problem is the 737 Max was hustled quickly to market to beat airbus and especially to offer a "drop in replacement" to existing earlier generation 737. Boeing would have definitely lost market share if they hadn't gotten their plane out then.
Choosing technical debt involves economic logic for today's marketplace. I suspect the real problem is companies wind-up deluded by their choice - to hustle the 737 Max to market, Boeing had to pretend is wasn't a, uh, piece of crap. Or not a piece of crap but a terrible mix of high tech and low compromises. And this pretending had to go through the whole enterprise. The marching order couldn't be "we'll add few more stiches on this Frankstein and hope the guts don't spill out", they had to be "this will be one more product of Boeing excellence" and everyone either believes or acts like they do. and this situation, combined with the huge investment involved, makes cutting losses very hard.
The software was fine (or at least not to blame). Unstable aeronautic design and lack of redundant sensors was the real problem.
Former engineering powerhouses like HP, Boeing, and IBM slowly get hollowed-out into incompetence when it’s assumed that Engineering and technical skills don’t require a considerable depth of knowledge and experience.
could it also be the normal generational rotation of people through the organization?
Everyone grew at a tremendous rate since the 1960's. Many, many more engineers. And then they retired.
The most effective leadership for a technical company is leadership with actual technical creds to back it up.
The technical side is a must because you need someone who actually understands the business and the innovations that must takes place. But you also need someone who can provide the people-leadership skills and the business wheeling&dealing.
These are all crucial skills and take up a persons time, so it’s not uncommon to see this come in the form of a duo (Apple and Steves, or Microsoft with Bill+Steve).
This is an engineering boondoggle and an embarrassment for Boeing. Software does not need to be this complicated. Design a damned airframe that's airworthy without needing stabilization hacks.
Boeing has to come out and say that the plane won't be back until at least 202x to take the pressure from their engineering and testing teams, so they can salvage this plane (and it may not be salvageable), instead of doing this month-to-month thing where they are hoping the FAA will sign-off on whatever latest change they put out.
They already let fly a dangerous plane. Given they have already done so, an explanation like 'stock market pressure' is no longer absurd.
Not to mention what happens if one of re-certified planes falls out of the air. People will refuse to board these planes.
But to your point, it makes up 0.78% of the S&P500 which is not insignificant for sure.
At least two of the planes you list are not being phased out because they were quads. The A340-300 was doing very well against the 777, however the A340-600 did very poorly against the 777-300ER. This is because the frame was too long and narrow, requiring extra reinforcement not to bend, thus ending up much heavier than the frame of the 777. The frame itself was grossly overweight, the extra engines were a rounding error.
The A380 was (and still is) extremely efficient (in terms of fuel burn per seat, at equivalent seating density), with a fuel consumption similar if not better than that of 787s or A350s, despite engines one generation older. It is, however, way way too big. In the end flying empty seats is very inefficient. But that has little to do with having 4 engines instead of 2.
The re-engined 777-9 will certainly be the most efficient plane flying when it is launched in 2021. Yet sales are lacking. It may, like the A380, turned out to be too big.
[1] https://leehamnews.com/2015/12/11/bjorns-corner-twins-or-qua...
So, de facto, each engine is strictly required, and having twice as many as a twin, quad jets are half as reliable as a twin, all other things being equal.
(me: licensed dispatcher who worked with the BAE-146 for a time. Incidentally, "BAe" stands for "Bring Another Engine" <grin>.)
>fitting 4 engines
Changing the number of engines would most likely require a total redesign of the wing.
Boeing certainly bears culpability for failing to execute, and whatever terrible lapses in certification were made to get there.
But why were they doing a dumb thing in the first place?
Because American Airlines asked-told them to. (Because they didn't want to have to pay to retrain pilots)
Boeing deserves to have substantial portions of management jailed over this, but there's blood on the legacy 737 operators' hands as well, for asking them to do it in the first place.
This sounds like the ratings agencies during the last recession, where they knew they were handing out good rating for junk bonds because if they didn't do it they'd lose a customer to the competition.
It's totally fucked.
Boeing was caught flat-footed and underinvested in a new product in 2011.
So while Boeing management could have said "No" to their largest (?) customer, that would have been a hard decision to make. And probably would have led to the board chopping C* heads for breach of fiduciary duty.
The Airbus order backlog is almost a decade out as it is, so American just shoving it all into A320neos that would take longer to show up wouldn't exactly be realistic either.
Boeing should've just gone with a clean-sheet design.
Hindsight is 20/20. The CEO chose a risky payoff (Boeing remaining competitive with the Max) over certain loss (Boeing losing many contracts to competition).
If the CEO didn't expect jail time (which given our history of such incidents is unlikely), he chose rationally in his own self interest.
If you want a different outcome you need to disincentivize this kind of behavior. E.g. if CEOs "taking responsibility" (because that's usually the reasoning for their insane income) meant they get locked up in prison without parole for 20 years. Then rational actors may become more careful.
Of course every customer ever is always going to ask for products on a faster timeline at a cheaper price. That doesn't absolve the manufacturer of its responsibility to build a safe product.
And I'd be inclined to agree it's 100% manufacturer fault... except this was a pretty high pressure ask.
IMHO, for me, it crosses over the line into sharing some of the blame.
Yeah, because American doesn't deserve blame for this nearly to the extent that Boeing does.
In December 2010, Airbus launches the A320neo, featuring new engines (LEAP included).
In July 2011, American Airlines releases a press release [1] containing the following:
"As part of the Boeing agreement, American will take delivery of 100 aircraft from Boeing’s current 737NG family starting in 2013, including three 737-800 options that had been exercised as of July 1, 2011. American also intends to order 100 of Boeing’s expected new evolution of the 737NG, with a new engine that would offer even more significant fuel-efficiency gains over today’s models. American is pleased to be the first airline to commit to Boeing’s new 737 family offering, which is expected to provide a new level of economic efficiency and operational performance, pending final confirmation of the program by Boeing. This airplane would be powered by CFM International’s LEAP-X engine." (emphasis added)
In August 2011, Boeing announces the 737 MAX program, featuring LEAP engines.
American Airlines literally ordered a plane that didn't exist. And then Boeing tried to build it.
There are structural failings (e.g. why Boeing wasn't better positioned by investing in a redesign in the 2006+ period), but American shares a fair amount of blame for this clusterfuck.
Boeing certainly could have said "No", in which case American likely would have bought additional planes from Airbus. So American had leverage, they used it to pressure Boeing into building what they wanted, Boeing failed at delivering that, and we're here today.
[1] https://www.sec.gov/Archives/edgar/data/4515/000119312511191...
Previous to American's press release, Boeing's leadership was strictly avoiding mentioning an up-engined 737. And was fairly reliably mentioning a clean sheet redesign program.
I can't think of a reason for doing that, if they in fact had such a program.
It would make sense to be trumpeting it loudly to anyone and everyone.
In effect you are stating that you understand better than the hundreds of engineers involved with this project why these design trade-offs were made. Careful with that line of thinking. You're nearly guaranteed to be wrong.
There are a host of reasons for the design in place: Efficiency, ease of implementation, familiarity with the components/technology, reliabilty of subsystems, availability of components, cost, maintenance complexity, etc. etc. etc.
And, don't forget, efficiency standards. As the whole world freaks out about CO2 and global climate change/cooling/warming, and the insta-crowd 'air travel shames' those who use these magnificent machines, it's important to realize that some of these design decisions maximizing efficiency are the hereditary descendants of cultural pressures, too.
Not to excuse any engineering f-ups, but there's a lot more to it than just 'build a simple tube with wings and an engine.'
While your line of reasoning is sound in most scenarios, in this case, the why is pretty obviously written on the wall. The overriding reason why this hacky-hack software solution was slapped into the airframe was to keep from having to reclass the plane following the addition of too-large engines. Your reasons listed played into the design decisions that became MCAS, but the underlying reason as to why any of this nonsense was needed in the first place is because non-engineers forced a shitty situation on the people who actually implement these airframes.
Hacker News is frequented by engineers from multiple disciplines, including software engineering, and many of those engineers know the kinds of terrible design decisions forced by management that come about to make a quick buck. People are pissed about it due to this fact, and rightly so.
The other half just think we're "pissed off at Boeing" and makes claims like "well, they should have just built the plane right."
And so you get engineers taking umbrage (IMHO, rightly so) for the ridiculous simplification being made by the second half.
We don't know that any engineers approved this design. It could have been completely compartmentalized and approved by management only. We'll never know what the engineers said until there's a complete investigation and the engineers are subpoenaed.
Here's what we know so far about the 'engineers': 1) The MCAS software was outsourced. 2) MCAS as originally designed and submitted to the FAA didn't have enough authority to affect the plane in the needed amount, so Boeing increased it's authority and never told the FAA.
Point to speaks to a failure in basic engineering or a deliberate deception of the FAA. A competent engineering team should have been able to calculate exactly how much effect was needed before the plane left the ground. Sounds like they were winging it on this one.
There should be way more strict user interface requirements. Including how humans are supposed to react to failures and whether that makes sense from human psychology point of view
If the MCAS had informed pilots of a potential sensor failure, then there would have been a very good chance that the pilots would have been able to land the planes safely.
Perhaps on the two crashes. We don't have any data on successful landings with MCAS disabled in very adverse flying conditions.
What if the plane is unsafe to fly without proper MCAS operation?
There should be a test that results in a plane crash with 50% of the simulations with MCAS behaving properly (IE, extreme weather, low fuel, land or die). Now during that same simulation, disable MCAS randomly. Also, there seems to be potential for the MCAS to come in and out based on disagreement (loose sensor wiring, etc), and see what the crash rate is for intermittent MCAS failures.
It's clear Boeing didn't do any of this kind of testing because they would have easily caught the failure modes that caused the plane to nose-dive into the ground. Completely predictable based on the behavior of the system.
When the purported self-regulation of a stock market company stands between making large profits, guess which one budges?
The idea that MCAS is some kind of 737 emulator intended to keep the plane under the 737’s certification, and that without MCAS it would have needed more thorough training, is a persistent myth on HN.
The reality is that MCAS exists because without it the 737 MAX’s aerodynamics are uncertifiable no matter how much training pilots were given.
During testing the MAX was discovered to have an inverted force curve on the stick approaching a stall — as the aircraft approaches stall angle, forces flip around and it becomes easier to pull the stick back (into the stall) than push it forward (out of the stall angle).
This violates a fundamental airworthiness requirement — commercial aircraft cannot be certified as airworthy if the stick forces invert.
MCAS “solves” this by commanding the stabilizers down when it thinks the airplane approaches stall — effectively using the stabilizer to put the forces on the stick that are “missing” due to the MAX’s aerodynamics, ensuring the force curve never inverts. MCAS isn’t really allowing the plane to pretend to be anything other than “a plane that doesn’t want to stall mid-air”.
Desire to keep the MAX requiring minimal training for 737 pilots (probably) drove the concealment of MCAS mentions from the manual, but MCAS would need to exist even if the airframe had been totally recertified from the ground up.
If I read this article correctly that was already the case.
So claims Bloomberg. Since there seems to be overwhelming consensus in this thread (that I also agree with) that it would be colossal stupid, I'd wait until we see confirmation from some other source. Bloomberg hasn't had the best record with facts or nuance in many stories recently.
https://www.seattletimes.com/business/boeing-aerospace/newly...
The 737 Max is stable, as all commercial aircraft have to be. MCAS is not a system to take an unstable 737 and make it stable. It's a system that was meant to take a changed 737 and make it fly nearly identical to a previous 737, so pilots didn't have to get a new type rating (a very big deal for airlines).
Juan Brown, a commercial pilot and certified flight mechanic, on YouTube has a (great series of videos)[https://www.youtube.com/playlist?list=PL6SYmp3qb3uPp1DS7fDy7...] talking through the mess that is Boeing made of MCAS, including what the actual problem is.
What changes have been made to the regulatory framework to prevent Boeing from signing off on their own safety tests? What changes has the FAA made to bring more public transparency to the flight certification process?
When is the sentencing date for Boeing executives? They are not getting a plea deal I hope?
I'm not worried about the overly complicated flight control software or MCAS, I'm worried about the next system that will fail because nobody at this company seems to care about engineering any more.
Are you talking about criminal court? Call me pessimistic, but I don't think our societal arrangement is such where the people at the top pay the price. I think regulation reform is a more realistic goal.
Obviously, continuous improvement is a core part of quality, and the FAA can always learn and improve their processes, but you can't expect a regulator to shoulder the core responsibility of certifying a plane. The primary responsibility is always going to be on the manufacturer because no regulator will ever have the manpower to test and verify everything nor the deep visibility into to R&D process that a manufacturer would.
The regulator should be adversarial; period. A well-meaning adversary, but adversarial never the less. Cutting manufacturers as much slack as has been is exactly what got us to the point we're at; a regulator that collected rubber stamped reports and only heard about things going wrong after tragedy has already struck.
It is better to have an active regulator able to intercede than to have the manufacturer coordinating everything internally, and asking for help when needed simply out of interest for removing the possibility to hide a problem discovery by never opening the floor to being questioned by the regulator.
If you tie the regulator's hands, then it isn't a regulator anymore. It's a postmortem service.
>but you can't expect a regulator to shoulder the core responsibility of certifying a plane. The primary responsibility is always going to be on the manufacturer because no regulator will ever have the manpower to test and verify everything nor the deep visibility into to R&D process that a manufacturer would.
In a financially incentivized market-based system where a fiduciary responsibility is built into the very underlying fabric of the corporate calculus, you cannot afford to be blind to the fact that a market actor has every reason and opportunity to stuff off every cost they can to improve their bottom line. This is why we need regulators in the first place, due to opposing optimizations between the interests of shareholders/executives, and the public.
The FAA as a regulator must be capable of requesting and having delivered any piece of information relevant to the goal of airframe certification. It is the job of the manufacturer to satisfy the regulator as to the objective safety of the plane, and it is the regulator's job to ensure nothing is left out for expediency sake. When regulator's start talking about streamlining things for the regulated, I start to get worried.
To go into more detail, no; I do not see the FAA adopting the actual physical task or logistics of testing a plane; however, I do see them as the final authority in terms of "Is the design complete" and "is your testing sufficient?"
This means that an Engineer, free of the inherent bias that comes from being dependent on the manufacturer for their paycheck, and acting in the public's interest as an external agent, needs to be as fully briefed on the entirety of the operating and physical details of every plane. It doesn't need to be the same person with it all; but the point is between the FAA as an external agency, and the manufacturer, there should be two independent agencies with enough understanding of the product that it can be demonstrated the manufacturer has done their due disclosure in informing the flying public of every facet of the aircraft's behavior that nothing like the MAX boondoggle should ever even be considered as being a reasonable course of action ever again.
You had people inside Bboeing who couldn't understand why MCAS was the way it was. Given that, it is evident that the most important stakeholders in being fully informed (buyers and operators) as a consequence were also not informed before regulators cut Boeing loose to sell on the market.
In 2018, legislation was passed that made it even more difficult for the FAA to exercise it's purported authority so long as a corporate representative assured them the issue was being handled internally.
I do expect anyone in an oversight position to be capable of observing things within their purview; and in terms of evaluating designs, the tangible nature of the principles and forces involved with aviation should be conducive to clear communication and reproducibility between the manufacturer and the regulator. The difference to a business in a functioning regulatory regime is that the manufacturer should see it's job as revealing new ground to a regulator, and leveraging the regulator as the source of of friction that peels away any uncertainty from the design. This can only happen in an environment where a "no more secrets" approach to business is maintained.
This is a great example of how looking at the world through the prism of stats really limits your understanding of what's actually happening.
The fact that you find my post novel and shocking just shows you're not reading very much about aviation in the past few months.
TDD advocates (and I'm a fan) will be feeling smug, be that doesnt apply here - the issue is not the initial tests, but explicitly tests after the fact, on criteria that werent in the initial tests. Be it by oversight or deliberate choice, TDD is in the same boat here.
All of which underlines how hard complex software can be. Boeing made lots of mistakes, and many of us might recall happier examples from our past (new criteria, but a well-written suite passes it all with minimal effort), but such examples are selection bias - if we exclude the code we know is a mess, the remaining mix of did-well and did-poorly code looked GOOD before. (Here I'm generalizing from my experience and the war stories I've heard)
Which brings us back to the Waterfall vs Agile issue. We know that we generally stink at anticipating all the requirements. We also know that the better we do at anticipating those requirements the less likely we are to have a sudden spec change derail us (not because we can prevent the spec change, but because our code tends to work)
Anyone asserting that such problems are simple to resolve hasn't worked on enough such problems. We are learning, but this field is still in its infancy and we've not even finished understanding some of the earliest principles the pioneers in the industry laid out.
I say get the 797 out ASAP.
Sadly, from my limited understanding, due to the engine location change on the airframe, the plane naturally pitches up... making it necessary.
One thing seems clear to the external observer, this is a 737 in name only.
They also probably could have succeeded, too, if not for other systemic problems within the company.
You say Boeing had no choice, but one choice was to acknowledge the problems, aim to keep the type certification with additional training, and negotiate with its potential customers on that basis. There is no law of mankind or nature that says Boeing is entitled to a certain number of sales at a particular profit.
The 737 has a feature almost no other new plane have: you can order it with a internal staircase so people can leave the plane without a specialized airport terminal.
This feature requires the plane to keep its current overall shape, otherwise the staircase thingy would not work.
So to keep this feature, while putting a bigger engine on the plane, they had to move the engine forward, and change its shape too, otherwise the engine wouldn't fit between the wing and the ground.
Then THAT caused the necessity of MCAS.
So long story short: wanting to keep backward compatiblity with the 737 staircase, led to the engine hack, that then to maintain backward compatiblity with the handling led to the MCAS hack...
So, it IS a 737, in the sense they kept the staircase and to do that ended needing the MCAS...
The other option would have been abandon the staircase entirely and make a taller plane, this would allow a bigger engine with no handling changes, but although this would remain a 737 from handling perspective, it would not be a 737 from the airports perspective: it would need to remodel the airports to install bridges or purchase of ladder trucks.
I assumed the stairs were not longer installed, and the height of the 737 was just an anachronism.
Video for those that haven't seen them: https://www.youtube.com/watch?v=unZeusTrDX4
No that would not be the major problem. Specifically since practically any airport nowadays has the infrastructure to service such a plane. This was quite different 50 years ago.
The reason why they couldn't have heightened it was that a new type certification would have been required and Boeing wanted to avoid exactly that at all cost.
Even at the cost of 346 dead people.
https://www.boeing.com/features/2018/08/737max10-landing-gea...
Most jet airliners have a pitch up tendency. The problem with the MAX is two related issues...
First, FAA has a requirement that stick pressure must follow certain patterns - namely, the harder you pull/push the stick, the more the plane reacts. Due to engine placement, the MAX did not meet this requirement.
Second, Boeing needed to create the MAX with minimal (approaching zero) re-training of pilots.
MCAS was supposed to meet both needs - by having MCAS adjust the stabilizer, the stick behavior/feel came back within requirements. And because it was all automatic, the pilots didn't need retraining in the simulator. Correcting the stick feel without a system like MCAS would have required more substantial changes to the plane, which would have likely required simulator time, which was likely a deal-breaker for some airlines.
“ As Boeing and the subcontractor that supplied the flight-control computer, the United Technologies Corp. division Collins Aerospace Systems, worked through these changes, it has at times created tension.”
This is an entirely normal and long-standing practice.
They didn't work on the Flight computer supposedly. Just the Flight displays.
There was a Bloomberg article with more detail.
https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...
> At least three Ryanair Boeing 737s have been grounded due to cracks between the wing and fuselage but this was not disclosed to the public, the Guardian can reveal.
> The budget Irish airline is the latest to be affected by faults in the “pickle fork” structure, which has sparked an urgent grounding of 50 planes globally since 3 October.
[1] https://www.theguardian.com/business/2019/nov/06/boeing-737-...
I'm saying this to highlight that what you're pointing out, while not desired, is kinda expected. That's the point of the routine inspection and maintenance - to catch these.
It's not the same as the conversation we're having under the OP: catastrophic failure due to bad assumptions in software (in this case, memory safety)
If it was expected, it wouldn't be newsworthy. And metallurgical issues in Boeing planes can be as critical as software issues:
> And metallurgical issues in Boeing planes can be as critical as software issues:
This much is obvious, and it's exactly why the maintenance exists. Crashes happen; the precursors are baked right into new processes and procedures. When said processes pick up on similar symptoms in the future, it might be in the news for a day or two, and then it disappears because it's expected.
The precise problem with the MAX is that there's an entire body of knowledge around the MCAS and other automation included with the plane that was never shared, which meant that unlike metallurgical issues which in many cases are largely unforeseen, Boeing's problems here were entirely preventable.
Items that stay in the news for a while tend to be the novel things.
At least on three occasions, multiple people have died or were injured because 737 NG planes developed serious cracks after exactly 8 years in service. Nothing like this is expected or considered to be normal. Therefore, when 50 planes of the same type are urgently grounded by multiple airlines in a very short period of time, it's considered to be newsworthy.
> The precise problem with the MAX is that there's an entire body of knowledge around the MCAS and other automation included with the plane that was never shared, which meant that unlike metallurgical issues which in many cases are largely unforeseen, Boeing's problems here were entirely preventable.
The previous metallurgical issues in 737 NG were also entirely preventable:
https://www.aljazeera.com/programmes/peopleandpower/2010/12/...
That makes the real source of any new issues with 737 NG questionable.
My point is, that MAX and MCAS is not the first instance of Boeing's negligence. And new things that went wrong with 737 NG might still be discovered.
The flying public in the US will naturally start asking questions. I guess airlines could offer discounts for Max flights. Then we'll see what level of discount is sufficient to get people on a plane that historically had a strong predisposition to nose-dive.
This will, as the article implies, come in combination with the justified and rightful concern that Boeing is and will continue and increasingly get more desperate and frantic to "fix" things (literally and, likely increasingly figuratively), as far greater testing and auditing requirements are placed on recertification.
I don't know about everyone else here, but I sure as heck am going to be quite careful not to fly on any MAX for quite some time if they are even ever re-certified again. The replacement for the 737, which the MAX was supposed to delay, was not scheduled to reach market until the 2030 timeframe (which could mean 2030 or 2039). I think it is anyone's guess whether Boeing has the resources to drastically accelerate that timeframe or the MAX village fire is draining all resources and it may delay that 2030 timeframe.
Please convince me otherwise, but I could see this MAX issue essentially crushing Boeing as it eats away at many different aspects of the enterprise over time. How long can you keep the concerns at bay and run on inertia? I think there may be hell to pay next year if this isn't really an "easy" issue that just takes some time and Boeing can scathe by. Maybe someone with deeper insight into Boeing's operations can substantiate why I am totally off base or … hopefully not … validate that my concerns are not unfounded.
> “It’s really complicated,” John Hansman, an aeronautics and astronautics professor at the Massachusetts Institute of Technology who is not involved in the repair, said of revising aircraft software. “It totally makes sense why it’s taking longer.”
But then...
> While the fix became more complex and politically charged after the second accident -- the crash of an Ethiopian Airlines jet on March 10 -- the changes to MCAS remained self-contained and relatively simple. “I could have a bunch of graduate students and rewrite MCAS in a couple of days and be done,” Hansman said.
So is it "really complicated" or "relatively simple" to the point where "a bunch of graduate students [could] rewrite MCAS in a couple of days and be done"? It can't be both.
"bunch of grad students"="patch MCAS to not auto-crash"
"really complicated"="redesign entire compute architecture to be multi-master and failure tolerant"
It was not clear to me whether it would be possible, in theory, to fix MCAS alone, and if so, who decided that it was necessary to go beyond that - Boeing? The FAA? Other countries' regulators? The airlines? None of these entities want another round of "you didn't fix it right", though their perception of risk and tolerance of it may vary.
AFAIK from this article, none of the earlier generations of 737s will be so modified, even though their systems are presumably very similar, in all non-MCAS aspects, to what the MAX first flew with. That's quite arguably a rational choice, once you recognize that there is a cost - benefit tradeoff even in air-travel safety.
This is a sort of technical debt situation. Ironically, if Boeing had not tried so hard to game the regulations, MAXes could be flying today with less fault-tolerance than they will have when they return.
But the article reads as if new hardware was necessary as well, which might change the picture.
I thought the whole point was that you needed 3 computers monitoring each other because if you have only two, you can't tell which one is faulty and which one is correct. How can they make it work here ?
So now I am curious. How do other airliners (newer Boeings, Airbus, etc) fare when subjected to the gamma ray test? I would like some context.
You keep CRCs of data in critical areas of memory and constantly check them to make sure you are only processing valid data, or use equivalent of ECC memory.