The programmer behind the Therac-25 fiasco was never found

However, it seems notable that they couldn't find the one person who actually knew the details of this situation. I suppose that in that era people could just move to a new place an be extremely hard to track down, but that ability to 'move away' and be hard to track down could also have been used as an attempt to lead to a more beneficial settlement. If this anonymous programmer were to testify that 'management was aware' of potential problems in the software, but insisted on shipping it anyway, it is the difference between negligence and gross negligence. The potential punitive damages would be dramatically increased in the latter case, and it would be 'cheap' to pay someone to hide to avoid this.

But that’s not the same thing as not knowing exactly who were present or involved when things happened, nor is it the same as not obtaining their assistance and testimony in reconstructing what happened.

This person should have been tracked down, 100%. Then having tracked them down, the report should avoid blaming them. It might even be useful to anonymize them after obtaining their testimony, to avoid harassment.

But if we aren’t interviewing them just to avoid blaming them, we have a very, very big meta-problem that needs to be solved.

And everyone, not just the best people, know this. Which is why, in a blame-oriented culture, everyone will just sit back and watch a colleague walk off a cliff. If you're not already involved, then getting involved risks painting a target on your back.

- Wishful cost and schedule "estimates" as a precondition for project approval.

- Feigning of current technical expertise as a job requirement for management.

- Hiding of real issues by lower management, for fear of a "shoot the messenger" reaction by upper management.

- Regulatory capture.

- The allure of the "plausible deniability" defense, ever-rising with ascending level of management.

The highest levels of management involved are of course boards of directors, and national legislators. And when was the last time any of us heard a "mean culpa" from either of those?

I think you are right that I’ve never heard about it in any other contexts though.

Not only that, but running in to the interlock should be considered a notable event. The machine should not just continue operating as normal, it should be clear to the operators that something potentially dangerous has occurred and should be investigated.

It seems like they had just assumed that because no one had managed zap someone with the previous models that the software must have been perfect, even though the previous models had hardware interlocks preventing the dangerous scenario. Those interlocks had presumably been tripped many times, just no one ever brought it to the attention of the vendor.

If a system trips a safety interlock it should fail to a safe configuration and remain there until reset by someone capable of investigating why it was tripped in the first place.

Modern traffic lights are a good example of doing it right. In those cabinets you see at every intersection, right next to the traffic light controller will be a device called a conflict monitor. This device will be wired to the circuits feeding the light heads themselves. If two conflicting movements are indicated for whatever reason, be it a failure of the controller, a short in the wiring, etc, the conflict monitor will trip and set the intersection to a fail-safe mode (usually either all-red blink or yellow blink for a main road with reds elsewhere) until manually reset by a human.

---

> I'm also always amazed by how few people seem to know about the Therac-25 incident, especially people that work in therapeutic radiation roles (in the UK anyway).

That's interesting, at least amongst my "techie" friends it's common knowledge. Many of us who went to college for computer science related things had it used in one class or another to get across the point that bad software can kill people in really unexpected ways.

I guess maybe the medical side of things doesn't find it worth as much attention because they don't have as much to learn from it.

Agreed. I was a developer on a medical instrument where the decision to implement a hardware interlock was made only after we were years into development and months away from product release. Initially, the belief was that we could get away with software-monitored interlocks but then the lead HW engineer realized it would be an uphill battle to get UL to certify the machine as safe. UL & OSHA want to see "hard interlocks:" systems where opening an interlock immediately cuts power and eliminates the Hazard without anything else in the control chain.

The problem was that the easiest way for the hardware engineers to implement this: by cutting power to all the controllers, was also the hardest way to manage in software.

At any given moment, with dozens of motors and actuators processing commands, all I/O controllers could have their power cut when an operator opened the cover. This meant retrofitting graceful failure tolerance (previously our code expected that these events would mean a hard fault and a system shutdown) to every single I/O command, resulting in changes of thousands of lines of code, and months of work to implement and review the changes.

In the end it was the right thing to do: the machine was rendered safe as soon as an operator opened the cover while system software kept the non-affected parts of the machine running as best it could. As wolrah suggests in the sibling comment, this was an exceptional event: the only way to recover was to acknowledge the fault and put the instrument through a restart procedure.

http://sunnyday.mit.edu/accidents/Ariane5accidentreport.html

https://www.inf.ed.ac.uk/teaching/courses/seoc/2008_2009/res...

"Everything we know in aviation, every rule in the rule book, every procedure we have, we know because someone somewhere died... We have purchased at great cost, lessons literally bought with blood that we have to preserve as institutional knowledge and pass on to succeeding generations."

We forget that often process and procedures which we now take as "common sense" or "bare minimum" were introduced as a best practice exactly because someone somewhere made a mistake that would have been avoided with those procedures in place.

Another thing I think is important to bring up, is that in software teams we often think that procedures are in place for 'other' complacent people or inexperienced juniors.

The reality is that procedures enforce consistency. And that consistency is needed for you as well not just for 'others'.

Today you write something elegant, tomorrow you could have trouble at home, be sleep deprived, pushed for a deadline, pressured by management and suddenly you in that instance become the 'others'.

Procedures in the end eliminate any wiggle for negotiable 'business compromises' or relaxing quality 'just this once'.

We agree that management was faulty. So why do we give them the presumption of good faith by taking their word for how the programmer operated?

We haven’ heard from the programmer, just from people who are covering up their identity. I’m not saying the following is likely, but it’s possible:

What if the programmer argued with them that they needed to do more testing and allow more time for development? Or that their should be a budget for a redesign, rather than cobbling the -25 from the bits and bobs of the -6 and -20, but management rushed it into production over their objections?

Then people die, and the programmer quits.

Management goes on to settle while being careful to make it impossible to talk to the programmer, who may very well have a lot to say about management.

We agree that management is at fault. Why take their word for it that the programmer operated without documentation? Why take their word for it that the lack of testing was the programmer’s choice?

Maybe the programmer produced a huge document explaining why the product should not be shipped, and management buried it to save their own skins?

I worked at that time in the industry. Half the people I worked with were musicians getting some extra dough by programming.

Bugs happen, but reasonable steps should be taken to prevent them. Developers share that responsibility with management, whether they like it or not.

Edit: It seems there was, as in he was developing alone and the company didn't disclose his identity. Puzzling.

But it would be nice if an investigation was completed, and all the facts found by the FDA before it was concluded. Maybe they could have collected information from him in a non-blaming way that would avoid similar disasters in the future.

The investigators did the right thing, they focused on the systematic problems. And the coworkers also did the right thing by not providing someone as a scapegoat.

I think it's also important to name him, to interview him. To understand how he came to kill. So that anybody writing life-critical code today can say, "I'd better not end up infamous like that guy. I can't make the same mistakes."

Maybe if he had been held accountable, the programmers at Uber wouldn't have been lax enough to code up the negligent homicide of Elaine Herzberg.

Exactly. Look at Experian. Attempted to blame an individual engineer for not applying a patch, then the next thing they know, the entire world is asking, why is your CISO a music major?

The software was badly broken, and had been for years. The difference between the 25 and previous models is that previous models had interlocks that did not allow the software errors to manifest.

At my first job we even had a separate safety officer for our low powered sources used for tracing waterflow.

Nobody says it was; such disasters are often multifactorial. But given to his position that person holds key knowledge and insights into what went wrong that no-one else has. Without access to that information, investigators can only hypothesize.

This is why things like whistleblower laws and indemnity insurance exist, to enable the full and unvarnished truth to come out. How are errors meant to be fixed correctly when you don’t have all the information as to the cause?

Compare how air crash investigations work. Or research into procedural improvements to hospital hygene. There are things far more important than just finding people to blame.

And hospitals and doctors alike do get sued for medical errors and negligence - especially lethal ones. "But I saved 99 other lives" is not really a good argument, it's not a game of points.

It was. There was a PE for software engineering until relatively recently in the US but no one took the exam because it basically wasn't required for anything.

Be careful what you wish for though. The requirements typically include a formal degree and some number of years working under a PE.

There's nothing magical about such a certification though. Other than the education and experience requirements, it's pretty much a GRE-type exam. I took the engineer-in-training exam way back when in a different engineering field but I stopped practicing before I sat for a PE.