It’s the spec bugs that kill you

> Nonetheless, the official said the incident shows that the Air Force and the Army have a serious training problem that needs to be corrected. “"We need to know how our equipment works; when the battery is changed, it defaults to his own location,"” the official said. "“We'’ve got to make sure out people understand this.”"

I hope that's not the mentality of the military today, that special forces operators in the field need to remember UI/UX details of the software they use...rather than the software developer adding in relatively simple safeguards, such as confirmation boxes.

It's also a lesson in how software development and engineering can go awry without close feedback from the audience. There are on-the-field realities -- such as the battery going low -- that even if the developers are aware of, they can't easily predict how the intended user will react in those scenarios...to tragic results, in this case.

As an example, consider the Therac-25 incident. It was a radiotherapy machine designed to operate in two modes - direct exposure to a low-power electron beam, or firing a high-power beam at a set of targets to produce X-rays. The predecessors used a loosely-coupled system - when the targets were not in place the high-power mode was electrically disconnected, or in other words the exposure system was separate from the control system. This was switched to a tightly coupled system, where the computer also served as the safety interlock. A particular sequence of inputs combined with a race condition could result in the targets (and sensors) being unlocked and removed but the beam firing at high power. Since the predecessors had hardware interlocks, this didn't result in any exposure. But they re-used the software going forward, and this bug surfaced.

https://en.wikipedia.org/wiki/Therac-25

No direct relation to this incident, of course. The controller should have a specific warning message and confirmation dialog before firing within a minimum safe distance. Probably wasn't in the specs though.

Stop.

Confirmation dialogs are dead UX. Users have been trained by internet browsers with pop ups and other poorly designed pieces of software with poorly worded dialog boxes to completely ignore dialog boxes. I've tested this several times. Users don't even consciously register that the dialog box ever appeared. You can watch them, right over their shoulder, and as soon as they close a dialog, they will turn to you and ask "what do I do now?" you will ask, "what did the pop-up box say to do?", and they will respond "what pop-up box?". Walk them through the process again, don't pre-warn them about when the dialog appears, and they will do it again.

This is also why crapware is so easy to install on any system that has wizard-based installers.

Do not use dialog boxes. It's better to hire two testing teams and set them against each other to break the software. Test and test and test again, then disallow bad behavior. The "plugger" example should not have allowed issuing a fire command on its own location, unless some explicit, completely out-of-band, one-time-use option would enable it so. Actually, better yet, it should not re-initialize the target location on start-up with the device's location. Just clear out the target, when the user tries to fire, they will see the lack of target and probably know "oh, must be because I changed the batteries". There is a reason they are called "fail-safe"s. You create failure modes that err on the side of safety. It's better if the device fails to fire than to fire at the wrong target.

I don't care if you can't afford it. If you cannot afford a large testing infrastructure, then you can't afford to make safety-critical software at all. You cannot solve this problem with software.

These situations come about when you have a management infrastructure that cares more about feature lists than correctness. I can hear them now, "we need to move faster", and "if it works, don't fix it."

I've seen both engineers and product managers use this as a crutch. And, frankly, as a PM I can say the blame generally lies with the product manager. This is our job. It is to understand how this technology will be put in practice by users and to make sure we are properly distilling and prioritizing the needs of those users into requirements for the engineers.

While I agree with the general point about specifications, we are talking about a weapon of war here. Interrupting a firing procedure for maintenance and then not checking the co-ordinates is exactly the kind of careless mistake I'd expect to get someone killed. At least it defaulted to the location of the receiver, preventing further errors, rather than a semi-random target.

I did not find anything in the official Washington Post archive: http://www.washingtonpost.com/wp-adv/archives/front.htm

This search shows other articles by that author on the Washington Post site during that time, but not this specific one: https://www.google.com/search?q=%22vernon+loeb%22+%22kandaha...

However, I think this is one of those (many) cases where it is an "operator error" which a better design could have prevented.

In other words, I think it was a plain bug. Somewhere in the fire() function there should have been some check for:

     distance(current_gps_coordinates, target_coordinates) < min_safety_distance

In the article, I explain why this is going to be a bigger problem in the near future, and speculate that some simulation-based could, perhaps, do the trick. But I am not sure.

Any thoughts about that?