In college they made us study the case of therac-25 so we understood the dangers of creating too many errors in our software.
(If by "dangers of creating too many errors" you just meant "don't create too many bugs" then that's not really relevant to this conversation, and also totally the wrong lesson to learn from that incident. The lesson is, software will inevitably have bugs, so make sure you have redundant checks to minimise their impact.)
Edit: see replies, originally I wrote fuses, but hardware interlocks is more correct.
They mention that previous Therac machines had "hardware interlocks" to prevent this sort of issue - maybe those are the safety fuses you're both talking about? - but it says they were never included in Therac-25 in the first place, because the software was assumed to be so reliable, not that they were removed because they blew.
Are you sure this is the same incident? Do you have a link to a source about it?
Edit: This article cited in Wikipedia is much more comprehensive and mentions that Therac-20 did have these fuses, in addition to hardware interlocks. But they weren't disabled because the real alarms were drowned out by false alarms - instead they all represented real instances of the issue.
https://web.archive.org/web/20041128024227/http://www.cs.umd...
(Middle column of the 12th page of PDF, numbered page 29 of the publication.)
> The Therac-20 at the University of Chicago is used to teach students in a radiation therapy school conducted by the center. The center’s physicist, Frank Borger, noticed that whenever a new class of students started using the Therac-20, fuses and breakers on the machine tripped, shutting down the unit. These failures, which had been occurring ever since the center had acquired the machine, might appear three times a week while new students operated the machine and then disappear for months. Borger determined that new students make lots of different types of mistakes and use “creative methods of editing” parameters on the console. Through experimentation, he found that certain editing sequences correlated with blown fuses and determined that the same computer bug (as in the Therac-25 soft- ware) was responsible. The physicist notified the FDA, which notified Therac-20 users.*
> The software error is just a nuisance on the Therac-20 because this machine has independent hardware protective circuits for monitoring the electron-beam scanning. The protective circuits do not allow the beam to turn on, so there is no danger of radiation exposure to a patient. While the Therac-20 relies on mechanical interlocks for monitoring the machine, the Therac-25 relies largely on software.
I remember talking about it several times over my college career, and for good reason.
To this day when people start talking about software in the loop in a safety context I get extremely twitchy. I don't let that shit happen on my watch.
Today, I’d ask about more ways to push the responsibility away from the firmware to stop catastrophic failures. If the debugger pauses the CPU you can’t use the CPU to regulate things. I’ve worked on things where the failure mode can be explosive and it makes dev really harrowing if you can’t rely on hardware interlocks.
They often don't have to know how the nitty-gritty of the HW works. HALs are used everywhere, and some of them abstract a lot away. I've interviewed embedded SWEs that couldn't explain how the tri-state buffers work within GPIO.