Hacker News Reader

I am pretty sure the software here did exactly what it was supposed to do. There was no buffer overrun, no NullPointerException, not any other kind of bug. There was just software that reacted exactly as specified.

The problem is: those specifications were bad. It's a bad idea to just rely on one single sensor when you actually have two, it's a bad idea to write software that disengages on manual input counteracting its decisions, then resume overriding pilot decisions out of the blue with a delay of a few seconds again. And it's an absurdly bad idea to do all of this while not telling the humans that will have to operate the machinery in the end ANYTHING about how all of this works.

I totally get why Boeing thought all of those obviously bad ideas were "good" - because they helped their bottom line. It wouldn't be possible to completely hide the existence of the MCAS if there was a potential for a sensor disagreement that has to be solved manually by pilots, and if it hadn't been possible to completely hide the existence of MCAS, it wouldn't have been possible to get the new plane out on the market without requiring expensive re-training. And that would have made a very convenient marketing claim of Boeing impossible to make: "this new plane can be flown by all those pilots familiar with the old plane, no re-training necessary, just read these few pages on an iPad and you're good!".

But this clearly makes this disaster not a "programmers killing people" incident. If any "xxx killing people" classification is to be made here, it's "managers killing people".

Bad programming was not the sole or root cause but it was a cause, and people died. You can't punt on that unless you want software engineer to actually mean code monkey. The same way a building architect has to sign off on the design demands of a a client, a software engineer on safety-critical systems must evaluate the spec and consider the effects of edge cases.

While a SW engineer isn’t a code monkey, they are probably the last line in a chain that started with someone saying let’s bolt some oversized engines on this plane, and really even if there is a SWE who spotted the problem and pushed back, the only way to effectively make this software safe is to not need it, I.e design a new plane, and the only way to make that happen is to blow the whistle. So yeah a SWE could have done this but so could anyone involved earlier in the chain.

What do you suppose the spec was here and how did the Engineer neglect to fulfill it? These specs are very compartmentalized, you think the Engineer is responsible for the big picture understanding and gets to take it up with the VP? No.

Yea, at a company like Boeing. They compartmentalize the fuck out of the engineering perhaps a bit too far.

I do not think all blame should be removed from the programmers that easily. Programmers should think about the ramification of their code and not just blindly implement bad specifications which will get people killed. Now it is possible that Boeing is such a toxic place that any disagreement is silenced, but I do not think we should automatically give the programmers a pass just because it is a specification issue.

The question is who is going to blow the whistle because that is what it comes down to. It’s just not a simple case of doing a better job. It’s going to need someone prepared to lose their job. If it was a simple case of “hey boss I think something is wrong here” and they listened and acted I think that would have happened. Something must be culturally wrong I am guessing.

It is bad hardware design with a software bandaid rushed in to maximize profits.

I agree with you though; I have said for many years that the way we write software (move fast, break things and shittier languages and infra like js and node) will eventually kill more people than most other man made disasters. Simply because the push for cheaper juniors in places where they do not belong.

Less bad but still potentially life altering I see every day in healthcare, insurance and payments: software is so horribly patched together that money just gets lost or misdirected.

I am biased as I was raised by a pupil of Dijkstra and studied under 2 of them as well, but for me there are solutions; tla+, Coq and languages with static typing. Not because these are silver bullets in their own right, but more because you cannot ‘move fast, break things’ in them; you have to think and plan a lot more as refactoring is very hard. Companies like Amazon uses tla+ a lot so I wonder why it is not standard practice in any place where lives are affected (aviation but also money and healthcare).

Why is this a software problem? Even if there is a software “fix,” doesn’t it seem that this tells us more about regulatory capture than using software to patch over bad hardware decisions?

It's a bad mechanical design, Boeing didn't want to spend money on retraining pilots, they asked software to fix it, software fixed it so that it remained the same, software had a bug, training didn't incorporate new system, because that was the point of the software, to not require a new type rating.

I think this is process failure from start to finish. Any sufficiently complex system will have bugs in it, whether it's software or not. I think the other failures are much worse. You have to account for your system maybe failing and your pilots have to be trained to deal with this failure.

And yes, software kills people, but no software kills even more people. Just look at how airplanes, cars etc. have improved safety through assistance systems.

It is bad mechanical design and the planes need to be renamed and rerated imho. That is now it should be fixed. It will cost Boeing 100s of millions but they deserve that by now.

I really didn't want to flame the room. As a commercially-rated small plane pilot and coder, I have a passion here and it both hurts me and makes me really angry.

I said this was a programmers problem, not a software problem, and I identified one of the key causes as programmers being isolated from user outcomes. Probably not a big deal if your user can't get to the website for 10 minutes. Different thing entirely here.

So this is a complexity * project organization problem. Yep, I think there's regulatory capture, and it may or may not have added to the problem. But you could repeat the factors I've listed in any country or industry you like and get the same results. So this isn't a "Boeing has corrupted the FAA" thing. Sure, it might have, and that's a good thing to talk about. But that's not it.

In fact, if you look at it that way, you're just continuing to silo off responsibility. This is another way of saying "Let's create yet another component that has oversight of all these other components that interact in ways we can't predict so that it can be the arbiter of what's safe or not" That's not a bad idea per se. It becomes bad because "the arbiter of what's safe or not" can mean all kinds of freaking things, many of which may be inconsistent or missing.

So really it just makes it worse. Let's say you make changes A, B, and C. Then it happens again with other systems. We come back in five years with another crash and sure enough, we've got the same clusterfuck as everybody points at one another saying that they've done their job but the other guy screwed up. Is it possible to black box test extremely complex systems? A lot of engineers have pretended that for a long time that you could. We are reaching the limit of how politically plausible that is.

I don't have an easy answer to the question "What do we do right now as programmers?" but I know that programmers, as a group of people, are in a unique spot to understand this problem and try to get attention for it.

A lot of us have worked on subsystems that eventually got used in ways we might find horrible and immoral. In that case you make a judgment call on whether the overall good beats the damage you may cause. In this case, however, these components and systems were built and integrated for no other purpose than this particular airplane. And bad things are happening. Joe Programmer can't be responsible for fixing Boeing, but he sure as hell can be responsible for having Boeing and other entities that build complex systems understand the horrible results of what they're doing and why.

ADD: Look at the comments here. It's a software thing. It's a hardware thing. Yesterday it was a training thing. A few days ago it was a distracted crew. Now we're talking about regulatory capture. Systemic problems are problems such that each piece can work in good faith and do a good job -- yet the overall results of the system gets worse and worse. A good sign of a systemic problem is when intelligent, well-meaning people end up pointing fingers all over the place. This is not a programming problem because it's in the program. It's a programming problem because we're in a spot to understand what got us here -- and I believe we have a moral responsibility to help others understand. A second sign is when new components are suggested as answers to problems the existing components are manifesting.

I agree with much of what you say here. I don’t agree that programmers are at fault or are particularly well suited or intervene, since the core is not a software problem. This is the nature of systems problems. Nevertheless, I think that we have a core agreement.

I wrote this two years ago -- the first chapter of a book (which I'm trying really hard _not_ to pump here)

https://wiki.info-ops.org/index.php/Main_Page#Info-Ops

There's a link to the first ten pages.

I find it difficult to write an intro like that and then see this happen again.

How can we be sure that this is programmers killing people, and not merely a busted angle of attack sensor, or an inadequate specification for how the plane should be designed (number of sensors or sensor robustness) or how the software should behave? This conclusion feels premature.

It's to early to conclude it's only a software bug. And blaming programmers is the absolutely not the way to avoid issues. Humans makes errors, always. Programmers or not. Some errors are unfortunately dramatic (talk about this to surgeons for example).

Now to remove the human factor as much as possible, there is processes and engineering techniques. If the failure was indeed caused by the software, and not by other defects in the plane design, the real question is how could it pass any safety check. For me, in any scenario, the first really wrong thing is how the plane passed all safety checks without the defect being detected, internally at Boeing as well as externally.

As more information unveil, it seems that the prime suspect is the pressure to release a plane too quickly.

> There you have it, folks, programmers killing people

In fact it is more like "unscrupulous capitalists killing people", though. The more we know about the incident the clearer it becomes.

> There you have it, folks, programmers killing people.

Programmers were killing people since before we had electronic computers.

IBM did punch-card system for mechanical computers for managing people in nazi death camps.