Hacker News Reader

Comment by mannykannot | Hacker News Reader

This article repeats the claim that the MCAS software was faulty, but is there any evidence of coding-level errors? There is the systems engineering failure of not comparing the two available AofA sensors to detect faults, if not going for at least triple-redundancy, but the root cause there is not a coding error.

It has been suggested that high AofA readings should have been rejected unconditionally, but that approach can have its own problems, as in AF447 where the stall warnings stopped as the airplane became more deeply stalled. One should not leave the decision as whether to do that up to the programmer's discretion.

View on HN

kind of depends on how you name it. my understanding is the code 'worked per requirements', meaning it did everything that it was supposed to, as defined by the requirements allocated to it.

for things like this, there is a layer of 'systems engineering', that decides (hopefully using an engineering process) how the airplane will work. this is codified into a 'specification', that is then handed over to software to implement.

if you are a software engineer given this task, you implement it per the specification. nothing more and nothing less. of course, if you have a concern, you bring it up (both informally and formally). there is no 'discretion' by the programmer.

The number of AOA sensors, how much control authority, how often the inputs are applied, are all decided by systems engineers (with aeronautical engineers input and analysis). the software engineer cannot 'second guess' if AOA failure has been considered and change the implementation.

The MCAS is a good example, why certain software must not be outsourced. Implementing the software correctly to spec isn't enough, you need enough understanding of the application to question the spec itself.

Software engineers are not aerospace engineers. I doubt the software people would have understood the aerodynamic effects of the MCAS system by looking at the requirements.

That is the problem with having pure software engineers implementing such a software. They need to have some aerospace engineering knowledge too. Or at least work directly in one team with aerospace engineers.

I am working as a "software engineer" in the semiconductor industry. By education, I am a physicist. I am in my job for more than 10 years now, and beyond my university education, I have quite a grasp of electrical engineering. Furthermore, most of my colleagues are eletrical engineers. So in doubt, I can just walk over to their desk and discuss any question beyond pure software development issues. And some of the software developers here are just electrical engineers, which expanded their skill profile into software territory.

Doing business this way is quite more expensive than having software developed by a contractor, but it creates safeguards.

That is true. But please keep in mind that the whole MCAS fiasco was precipitated by a dubious cost-cutting initiative related to an engine placed more in front than would be aerodynamically stable. The game was lost in the first quarter. The whole point is that civilian aircraft, unlike fighter jets, have to be aerodynamically stable without the need for such software, whether developed by holistic engineers, or compartmentalised subcontractors.

The 737 MAX is aerodynamically stable. MCAS isn't even active while any flap is extended. MCAS should be active only at angle of attacks, which wouldn't be reached at extreme angle of attacks. The sole purpose of MCAS is to emulate the exact reactions of the older 737 machines in this domain, as this was required for maintaining the same type rating. That is also the reason, the pilots are supposed to deactivate MCAS via the runaway stabilizer checklist. This software isn't required for stable operations of the airplane - Lion air had a MCAS malfunction the day before the crash, but disengaging the electric stabilizers solved the problem and they were able to finish their flight without incidents.

Incorrect. The aircraft is not aerodynamically stable in pitch unaided by software, thereby failing the demonstration of positive pitch stability as written in FAR 25.173. The non-compliance is of the form of a slackening of control stick response force at high AoA due to extra lift generation ahead of the center of gravity caused by the forward mounted engine nacelles.

In regulatory parlance, this specifically means the airframe (the specific physical assembly of parts that collectively determine the flight characteristics of the object in question) possesses an "instability" that disqualifies it from being certified airworthy as a Civil Transport Aircraft unless the FAA amends or reinterprets the requirement.

The penultimate Lion Air flight only recovered thanks to The intervention of a third pilot; a luxury not universal to every cockpit.

>MCAS should be active only at angle of attacks, which wouldn't be reached at extreme angle of attacks.

The potential for MCAS subroutine activation is active during all phases of manually controlled (Autopilot off), flaps up flight,and is gated only by readings from an AoA sensor; which can fail in flight to catastrophic effect.

>The sole purpose of MCAS is to emulate the exact reactions of the older 737 machines in this domain, as this was required for maintaining the same type rating.

Incorrect. The sole purpose of MCAS is to induce an intentional "mistrim" during certain points of the flight envelope to counter the extra lift generated by the repositioned engines, with the end goal of smoothing out the non-compliant stick force response curve. Note, jury is still out whether this should even be allowed. I've gotten wildly different responses based on who I've talked to. Pilots seem to express horror or extreme discomfort. Aerodynamicsts are disconcertingly quiet, but tend toward hard reconfiguration to obviate the need for a software based solution.

Also, MCAS specifically deviates from oldaircraft behavior in that it removes an override of auto-trim commands activated by heavy back pressure on the yoke. There were also specific reconfiguration of the stabilizer trim cutout switches that removed any capability of locking out the MCAS system's ability to actuate the trim motor, while leaving pilot's thrim switches active.

https://www.seattletimes.com/seattle-news/times-watchdog/the...

https://www.seattletimes.com/business/boeing-aerospace/boein...

Make no mistake: Boeing screwed the pooch. Big Time.

some would, some would not. those that have been doing aircraft software for 20 years probably have a fair knowledge of airplanes, aerodynamics, and flight controls.

your comment expresses my feelings exactly on 'was this a software failure?'. No. But software was involved.

This is the larger problem. There are plenty of aero engineers that are brilliant programmers yet the reverse is a much smaller number. Add to that, aeros that can code are far more expensive than software engineers, it becomes easy to see why they outsource to software people.

That assertion in the article seems to be incorrect according to the linked Bloomberg article, which states that the MCAS software was not outsourced: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-.... Given that apparent factual error, I’m suspicious of the other claims, which are highly rhetorical and mostly asserted without evidence.

anything that is connected to the flight controls, is too important to sub out (unless you are subbing it out to someone better that you are).

making an airplane is a huge endeavor that crosses across many domains. you have to have competency up and down the 'stack'. Boeing does.

Maybe this will cause a re-look at outsourcing policies.

(edit. yes, this software was not outsourced. but it may add another dimension other than cost for future decisions)

It has been mentioned in the article that MCAS was never outsourced. Also it’s unfair to blame the software engineers who are building it to specification. It would be fantastical to assume that Boeing management( the same one ploughing through engineering with bean counting) would give a rats crap about a low level engineer raising a concern.

Not sure why this comment is being downvoted.

First, none of the software in question was outsourced to the company.

Second, the Bloomberg article does not describe the problems Boeing had with the software engineers from HCL, beyond a cryptic, "it took many rounds going back and forth because the code was not done correctly." Was it a genuine misunderstanding of ambiguous language? Was it because the programmers did not have a background in the aerospace problem domain? Or was it because the engineers had a poor grasp of the vocabulary? It seems improper to assume the third situation in the absence of any additional evidence.

Finally, I am not sure what part of "software being outsourced" is considered outrageous. Is there evidence that a software engineer employed by Boeing, rather than by company XYZ, is less likely to produce faulty code? Or is more likely to understand the complex downstream effects of their actions and the equally complex upstream design decisions which informed them? Also, countries don't write code. Companies don't write code. People do. That "India doesn't have a deep background in aerospace" doesn't mean much at the level of individual engineers. I am skeptical that the most highly skilled American programmer has a deep background in aerospace either.

Sorry about the rant: something about the Blookmberg article came across as xenophobic innuendo.

MSFT had a similar experience with HCL (I think it was them, one of the large Indian outsourcing firms in any case) when I was working on Vista. All fixes for bugs identified by static analyzers (PREfix/PREfast) were outsourced to contract programmers who had zero familiarity with the Windows source code. The “fixes” they sent back almost invariably introduced new bugs, and the whole back-and-forth process took much more time and effort (and presumably money) than just having the code owners fix the bugs in the first place.

It has been mentioned in the article that MCAS was never outsourced.

Where did the article say that? To me it explicitly claims it was outsourced:

the McDonnell Douglas influenced Boeing new one tried to patch the problem with software. And it was bad software, written by to engineers paid $9/dollar an hour.

And I am not blaming the software engineers for building it to the specification. In contrary, I clearly stated, that just building it to specification isn't enough for such critical software. Such software needs to be developed in teams where enough team members are senior enough to properly raise concerns where appropriate.

Nowhere in the [original reporting][1] does it say that MCAS was outsourced.

This article (The Coming Boeing Bailout?) wrongly asserts that MCAS was outsourced. There's no corroboration for that.

Mind you, I'm not on Boeing's side. I'm just clarifying what's been reported so far.

[1]: https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...

https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...

>In offices across from Seattle’s Boeing Field, recent college graduates employed by the Indian software developer HCL Technologies Ltd. occupied several rows of desks, said Mark Rabin, a former Boeing software engineer who worked in a flight-test group that supported the Max.

>The coders from HCL were typically designing to specifications set by Boeing. Still, “it was controversial because it was far less efficient than Boeing engineers just writing the code,” Rabin said. Frequently, he recalled, “it took many rounds going back and forth because the code was not done correctly.”

Nowhere in that quote does it say that:

1. HCL wrote the code for MCAS; or

2. HCL wrote all the code of the Max.

But again, I'm not defending Boeing. I think they made some (deadly) mistakes. But so far there's no proof that the MCAS software was outsourced.

I had understood the part of the article, which I quoted as that MCAS was outsourced. If it wasn't then the article is to blame and my point is moot.

> This article repeats the claim that the MCAS software was faulty, but is there any evidence of coding-level errors?

No, there isn't. The Verge (and IIRC other sources) have reported that the crashes were due to failures in the angle of attack (AoA) sensors. No program in the world is going to give you correct output with garbage input.

Verge link: https://www.theverge.com/2019/5/2/18518176/boeing-737-max-cr...

No sensor failure should cause the plan to go into swan dive to destruction mode.

The software that carries your tweets has to deal with exceptional situations including hardware failure so to must software that runs life critical hardware. Seems to me that if the proper response to faulty sensors was to disable MCAS perhaps it should have just done that for the pilot.

Even without a second AoA sensors perhaps something could have been derived from the fact that the pilot was furiously pulling up while the aircraft was losing altitude?

Defense in depth is always a good thing.

> Even without a second AoA sensors perhaps something could have been derived from the fact that the pilot was furiously pulling up while the aircraft was losing altitude?

People keep saying this. But AF 447 was caused by this. Pilot furiously pulling up, plane rapidly approaching the ground... And the steps to fix it would have been to push down. None of this is as simple as you seem to think.

AF 447 was caused by not being "ahead of the plane", or not being "10% smarter than the piece of equipment you're operating".

Pilots speak of being ahead of the plane as the mental state of understanding the current condition of the aircraft, and being able to correctly forecast the end result of any control action on the current state.

This is like driving a well maintained car. You know the car will "follow" your directions.

Reasoning "behind the plane" is a state where the plane has started to do something, and you have to figure out what changed; I.e. the actual plane is no longer equivalent to your mental model of the plane. You can no longer with any certainty forecast the end behavior of the plane on your understanding alone. You're essentially in a reactive state. You have to rebuild your model.

AF447 suffered a major automation casualty. The pilot's didn't realize this, and attempted to fly off their flawed mental models of the plane's behavior. For instance, the Stall alert becoming active when the pilot tried pitching the aircraft down being unsilenced due to the automation coming back within non-extreme input regimes. The pilot attempted to avoid the alert by returning to the extremis instead of pitching down through the alert to recoverable flight.

Did the pilot's know that the alert would silence beyond an extreme value? No, but instinct told them if they did something that caused a stall response, reverse it to get away. Without the extra intelligence around how the automation worked, heuristic reasoning led to a valid, but ultimately lethally unsound course of action.

No disagreement at all from me. I just wanted to point out that the suggested 'should I cancel MCAS' heuristic was not nearly sophisticated enough.

The software did way more dumb things than it needed to in the presence of these failure modes.

Those 'dumb' things were, in fact, the things that the domain experts specified it should do in response to the data that the software received.

Reply on news.ycombinator.com