Those who do not read the history of Therac-25 apparently are doomed to repeat it.
If the hardware can be destructive under the control of software then you must have hardware interlocks, not just software safety features.
If the hardware can be destructive under the control of software then you must have hardware interlocks, not just software safety features.
Also on that list (from memory).
* AT&T's switch cascade failure * Mars Climate Orbiter * Ariane 5
He also took us through some bugs in code he'd written when he was doing massive telecomm's systems.
I came away with a few lessons.
* If it can't fail, test it twice. * If it's designed so it can't possibly fail, test it three times. * All code is buggy.
When faced with this kind of requirements, I always lift my eyes right above my screen, where it is written:
> The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair.
And for emphasis, next to it is a poster of a nuclear island cut-out.
Sure enough, we once had a motor controller break where the problem was interconnecting wiring...
For those who haven't read it, http://sunnyday.mit.edu/papers/therac.pdf
1. Those who read about Therac-25 and decide never to work on any systems where bugs could threaten lives (your chosen course, and mine).
2. Those who read about Therac-25, get frightened, and work very carefully on such systems.
3. Those who read about Therac-25 and think, "Those idiots! Good thing I'd never make a mistake like that."
The prevalence of (3) in all professions and walks of life is kind of frightening.
If you dig into the details of why its programmers and their managers killed people, it's very easy to say "I'd never be that irresponsible", but still screw up with much more subtle bugs.
:)
But it can be stressful to write software that has a very direct impact on peoples lifes. While I was rolling out an updated version of an inhouse application a serious bug was discovered. The application was used as part of chemotherapy follow-up/planning and on some occasions it gave the wrong answer for a test. I discovered that the bug was old and that wrong answers had been given for a long period. Personally I was relieved that I had not caused the bug, but from a medical perspective it would have been better if the bug was new and had only affected very few patients.
My takeaway from that experience is that care should be taken when writing critical software, but also that we will make mistakes and it is very important to always focus on how we can make fewer mistakes instead of pointing fingers.
Even if obligation to publish source code were to harm companies (which might even not be the case) the benefit it would bring to the consumers and whole ecosystem is more important.
Companies don't need special protection. The most important advantage of a company is that it can die without killing all people who work there and destroying all its assets. This allows companies to take risks no sane person would take if he were to be responsible with all his money (both savings and potential debt).