Software Engineering Lessons from Aviation
riceo.me
riceo.me
For me it's the mindset that differs. Too often as software engineers we find a bug and just fix it. Aviation goes a step deeper and finds the environment that created the bug and stops that.
Unfortunately, the recent 737 MAX incidents seem to have changed this. From what I understand the reaction to the problems sounds more like what I'd expect a software business to do, rather than the airline industry!
Those that value quality are going to be swimming up stream in most organization that develop software because the bean counters always go straight to fast and cheap.
These are issues that will definitely recur in troubleshooting future bugs, and doing a proper postmortem could easily save 250+ man hours over the course of a year. What's more, fixing some of these issues would also aid in application development. So you're looking at immediate cost savings and improved development speed just by doing a napkin postmortem on a simple bug. I can't imagine how much more efficient an organization with an ingrained and professional postmortem culture would be.
John Chidgey digs into well known catastrophes, analyses what went wrong, and what was fixed afterwards. Not software related but promotes a safety mindset very well.
I think there's some nuance about MCAS that's lost in all the media reports. As far as I understand, the MCAS software didn't have a "bug" in the sense we programmers typically think of. (E.g. Mars Climate Orbiter's software programmed with incorrect units-of-measure.[0])
Instead, the MCAS system was poorly designed because of financial pressure to maintain the fiction of a single 737 type rating.
In other words, the MCAS software actually did what Boeing managers specified it to do:
1) Did the software only read a _1_ AOA sensor with a single-point-of-failure instead of reading _2_ sensors? Yes, because that was what Boeing managers wanted the software to do. It was purposefully designed that way. If the software was changed to reconcile 2 sensors, it would then lead to a new "AOA DISAGREE" indicator[1] which would then raise doubts to the FAA that Boeing could just give pilots a simple iPad training orientation instead of expensive flight-sim training. Essentially, Boeing managers were trying to "hack" the FAA criteria for "single type rating".
2) Did software make adjustments of an aggressive and unsafe 2.5 degrees instead of a more gentle and recoverable 0.6 degrees? Yes, because Boeing designed it that way.
Somebody at Boeing specified the software design to be "1 sensor and 2.5 degrees" and apparently, that's what the programmers wrote.
I know we can play with semantics of "bug" vs "design" because they overlap but to me this seems to be a clear case of faulty "design". The distinction between design vs bug is important to let us fix the root cause.
The 737 MAX MCAS software issue isn't like the Mars Climate Orbiter or Therac-25 software bugs. The lessons from MCO and Therac-25 can't be applied to Boeing's MCAS because that unwanted behavior happens in a layer above the programming:
- MCO & Therac: design specifications are correct; software programming was incorrect
- Boeing 737MAX MCAS: design specifications incorrect; software programming was "correct" -- insofar as it matched the (flawed) design specifications
[0] https://en.wikipedia.org/wiki/Mars_Climate_Orbiter#Cause_of_...
[1] yellow "AOA Disagree" text at the bottom of display: https://www.ainonline.com/sites/default/files/styles/ain30_f...
I always liked this quote from the "Mythical Man-Month": “Never go to sea with two chronometers, take one or three”.
https://blog.ipspace.net/2017/01/never-take-two-chronometers...
(I can't tell if you're making a side comment or specifically replying to the categorization of "software bug".)
Yes, the 737 MAX only has 2 AOA sensors instead of 3 like Airbus A320. This is a physical design of sensors mounted on the airframe. But this aerospace engineering design detail seems outside the scope of assigning blame to software programmers writing code. (There isn't a software coding methodology that makes a 3rd AOA sensor appear.)
For such an important system, I would think that a single instance would be too little (single point of failure: even in less mission critical system it wouldn't be allowed), while using two you would not be able to resolve a reading conflict between them (which is the point of the quote), so three is probably a reasonable number.
Even if applied to design and not to software programming, the concept is still sound. The point was what the quote meant, not its source.
It's why they didn't do it.
When reviewing a design, the first thing to verify is if it can satisfy its input requirements. In your example, a design that has to satisfy a safety requirement but doesn't is not correct and must be rejected.
Requirements are product features that must be present. A Design is one of many potential ways to satisfy that set of Requirements.
For example, a requirement might be "the user shall not be exposed to hazardous voltages (defined elsewhere) when servicing the equipment."
A possible Design solution might be "provide cover interlock switches so when the covers are opened, all voltage supplies are disconnected." or "software monitors a cover switch, and when that particular cover is opened, a command is sent to the power controller to disconnect power to anything that is reachable from that opening."
Which of the two (or other) design options is chosen, is a Design Decision, but they are means to an end, that end being Satisfying The Requirement.
> Somebody at Boeing specified the software design to be "1 sensor and 2.5 degrees"
What is called a "software design" there, you would probably call a "requirement".
I agree with you. Your meaning of "design" has a notion of correctness.
OK but how do we know, how is it demonstrated, that this financial pressure condition has now been mitigated? What is the exact nature of the "fix"? And actually what are all of the closed door conversations, back then and now, about the various possible behaviors for this software routine? How is it they came up with that one? How is it they come up with the new one? And really, why is the first one wrong (aside from the fact there are a bunch of dead people, which is a consequence of the original error)?
And which parts of the design? There are many parts to it. Not all of them are as bad as others.
As a pilot I find it impossible to imagine a closed door room with engineers not computing, let alone not imagining, the potential for this particular failure mode. And if a pilot were present in that closed door session, I find it impossible they would not immediately be bothered by the potential for mistrim at low altitude that would result in too scary a probability of unrecoverability.
It makes me wonder if pilots were even involved at that level of the design and decision making for the feature.
They were not.[1]
[1] https://www.wsj.com/articles/boeings-own-test-pilots-lacked-...
If you're a food products company, you bring in people to do some kind of product testing to see if they like it or hate it. So far I have yet to read or hear from a pilot who says: Oh this is GREAT idea! Love the entire concept!
This was not an engineering problem, it was a greed-created management intentional decision. It was management designed for failure because management changed the goal to put money over life in distinct ways.
- I Catastrophic - Death, and/or system loss, and/or severe environmental damage.
- II Critical - Severe injury, severe occupational illness, major system and/or environmental damage.
- III Marginal - Minor injury, and/or minor system damage, and/or environmental damage.
- IV Negligible - Less then minor injury, or less then minor system or environmental damage.
Now, face it, most webcrap and phone apps are at level IV. Few people in computing outside aerospace regularly work on Level I systems. (Except the self-driving car people, who are working at Level I and need to act like it.)
MCAS started as just an automatic trim system. Those have been around for decades, and they're usually level III systems. They usually have limited control authority, and they usually act rather slowly, on purpose. So auto trim systems don't have the heavy redundancy required of level I and II systems. Then the trim system got additional functionality, control authority, and speed to provide the MCAS capability. Now it could cause real trouble.
At that point, the auto trim system had become a level I system. A level I system requires redundancy in sensors, actuators, electronics, power, and data paths. Plus much more failure analysis. A full fly-by-wire system or a full authority engine control system will have all that.
So either MCAS needed to have more limited authority over trim, so it couldn't cause trim runaway, or it needed the safety features of a Level I system. Boeing did neither. Parts of the company seem to have thought the system didn't have as much authority as it did. ("Authority", in this context, means "how much can you change the setting".)
Management failure.
There are two other dodgy things going on. One you can't disable MCAS without totally disabling the electric trim. And the mechanical advantage of the manual trim isn't sufficient to re-adjust trim once it's too far out. And hasn't been _forever_.
(When pushed, they might say "That was a groundbreaking book, for its day, but the industry moved on." Now we've got open floor plans, and AGILE SCRUM, and free snacks ... and also no evidence these are an improvement to the software development process, but never mind.)
This aviation mindset you refer to is the same way. I can't tell you how many times this happened to me:
- User clicks a button, and it doesn't do what it says it should.
- A bug is filed, and assigned to me.
- I investigate, and find the problem. I start preparing a fix.
- Manager comes by to pester me. "Why isn't this button fixed? Shouldn't that have been a quick fix?" We played Planning Poker last week and everybody else who isn't working on it agreed it should only be a 1!
- See, we're computing this value incorrectly, and I grepped the codebase and it turns out we're also doing it wrong in 7 other places, which causes...
- "The customer wants this one button fixed. Don't worry about the others. Don't worry about testing, or cleaning up, or documenting why the mistake was made or how it should have been done. Those aren't on this milestone. Just fix this one button and move on. We need you working on the new features we promised our customers this month..."
Modern software development is a circus of improperly aligned incentives.
Sounds like getting management to truly understand the concept of technical debt is the core problem.
One thing I have not had very good discipline about is I want to use checklists both for code submitted for review and when I'm doing reviews. Lint checkers etc. can only go so far.
If anyone has published checklists for code reviews I'd be curious to see them. This one seems reasonable: https://www.liberty.edu/media/1414/%5B6401%5Dcode_review_che... though I'd add concurrency to the list.
> 1. Don’t kill yourself
> 2. Don’t kill anyone else
Could we reorder these, though? Every once in a while a plane will hit a house and kill its occupants (and the pilot, usually) and it's so awful. I think not killing others as a pilot is so much more important than not killing yourself.
If your job is to save a life and that life depends on you, you don't do anyone any favors if you die
The industry is really slow to change its practices and tools. Like the use of C for most software, I do feel a more safer language out to be preferred.
Use of 1553 bus for inter device communication, the bus and protocol aren't general, it is very opinionated/rigid about the manner in which communication should happen. And the hardware parts for it are horrendously expensive compared to most ethernet, IP equipment. There is an aviation ethernet standard, but adoption of it has been slow.
This could be a strong factor in its popularity. If things must happen in a certain order, then the behavior of the system becomes easier to verify. Ease of verification should never be understated in safety-critical systems.
And what language would that be, where it has absolute determinism (which rules out anything with GC)?
They tried using Ada years ago for avionics. The problem here is that no one knows Ada any more, and no one really wants to make a career out of it since it isn't used anywhere else.
So, in practice, C and (a narrow subset of) C++ get used. Maybe Rust would be a good choice in the future.
In an airplane, if things to badly, you keep flying until you land. If things go really badly, remember that everything is built to be light weight, and unless the crash is well controlled, everything will be destroyed and everyone will die. If your engine quits, your cabin ruptures, your instrumentation fails, you keep flying. And you need instruments; in poor visibility, your own sensory inputs are in fact faulty, and won't help you figure out which way is down.
Unlike in a car, where it's pretty obvious where the ground is, for example.