Therac-25
en.wikipedia.org
en.wikipedia.org
Especially funny was the formal verification course that mentioned the Ariane 5. Apparently all the new software in Ariane 5 was formally modelled and verified, but one part of the system was directly ported from the Ariane 4. Because the Ariane 4 mission had been successful they did not verify that (it's an expensive process). The bug that crashed the rocket involved the fact that the Ariane 4 was 16 bit, and the Ariane 5 was 64 bit, it resulted in an integer overflow somewhere leading to a crash.
You can spend millions in painstaking formal verification, and pay for the small part that you did not verify.
> The Ariane 5 reused the inertial reference platform from the Ariane 4, but the Ariane 5's flight path differed considerably from the previous models. Specifically, the Ariane 5's greater horizontal acceleration caused the computers in both the back-up and primary platforms to crash and emit diagnostic data misinterpreted by the autopilot as spurious position and velocity data. Pre-flight tests had never been performed on the inertial platform under simulated Ariane 5 flight conditions so the error was not discovered before launch. During the investigation, a simulated Ariane 5 flight was conducted on another inertial platform. It failed in exactly the same way as the actual flight units.
> The greater horizontal acceleration caused a data conversion from a 64-bit floating point number to a 16-bit signed integer value to overflow and cause a hardware exception. Efficiency considerations had omitted range checks for this particular variable, though conversions of other variables in the code were protected. The exception halted the reference platforms, resulting in the destruction of the flight.
Although the article partially disagrees with tinco: it looks like formal verification was only implemented after flight 501:
> The launch failure brought the high risks associated with complex computing systems to the attention of the general public, politicians, and executives, resulting in increased support for research on ensuring the reliability of safety-critical systems. The subsequent automated analysis of the Ariane code was the first example of large-scale static code analysis by abstract interpretation.
My understanding was the routine in question was used for re-calibrating the inertial guidance system in the Ariane 4 in case of an extended hold-down period for up to 40 seconds after ignition. Presumably this routine integrates measured acceleration, which can be divided by hold-down-time to find the average error in inertial bias over the hold-down period. The average error in accelerometer bias (in other words, rate at which measured ground-relative velocity deviates from true ground-relative velocity) would then be subtracted from the previous bias estimate in order to get the bias estimate to be used for the flight.
Edit: even though recalibration was never intended to be used in the Ariane 5, the integration routine was left in and continued to integrate acceleration measurements for 40 seconds after ignition.
The Ariane 5 is capable of undergoing more acceleration than the Ariane 4, so it was possible within 40 seconds of ignition for the 64-bit velocity (integral of acceleration) to overflow the 16-bit variable it was cast into at some point. With no range checks implemented for this cast, this routine caused the computer handling the inertial guidance to crash and dump. The autopilot saw the crash dump, but misinterpreted it as a position and attitude update. The autopilot then adjusted the rocket nozzles to correct for the misinterpreted attitude, causing the rocket to start flying somewhat sideways through the air at high speed, leading to breakup due to aerodynamic forces.
Most (all?) modern launch vehicles contain small explosive charges (or lines of detonating chord) to burst fuel tanks, break up the solid fuel grain, and perhaps break up some of the more dangerous pieces if the vehicle deviates too far from the planned flight path. Shortly after aerodynamic forces began to break up the Ariane 5, some internal automated system detected things were going very very badly and triggered the auto-destruct sequence.
It's obviously on a far different scale from software causing deaths, I'm not trying to make comparisons between the two. But the motivation of the software developer needs to be a little more moral than just 'will I be in trouble with the law?'
The kind of programmer that stores passwords in plaintext in a publicly-accessible database is not one that says, "I'm going to do this because it's faster." He or she doesn't even know that that could be a problem. Similarly, those hiring such a person are ignorant of what experience costs, and so they get what they paid for, even if they didn't make the conscious decision to make an insecure website.
(You don't see Google or Microsoft or Apple posting ads on freelancer sites for $5/hour jobs. They know the results they want cost more than that.)
It is not just a matter of developers' professional and personal integrity. We have to live with the consequences of bad software produced through the employment of amateurs, even if we, personally, care about doing a good job.
It is really easy to pick up languages and libraries with so much information being freely shared. IMHO that usually comes with less regard for proper rigor in exchange for fun and keeping interest.
Although, I think the vast majority of new programmers aren't working on things like Therac-25 machines, but instead on Flappy Bird, JavaScript library #749237, and Bob's Website. You don't really need much software design knowledge to push products at that level.
That is the way it has been since computers went commercial.
throwing together a website doesn't make you a hacker. allow me to rephrase.
using some tools a person built for you to throw together a website, and you using them together in exactly the manner that they were meant to be used makes you a user/consumer not a hacker.
quite a few of actual hackers i saw withered in university because they did what grad students did in their high school days at home.
i can tell you a lot of things that you use in production today even in your field is the result of a lot of these hackers you so disparage.
edit: rephrased last paragraph
Also note that the roles of "hacker" and "software developer/engineer" need not be mutually exclusive.
I was pointing out to the parent comment that someone who is not reasonably versed in proper software design, process, and documentation cannot just hack away at products such as Therac-25 anymore. At least, it would be much more difficult for them to do so and bring it to market successfully, due to all the regulatory red-tape.
why don't you rephrase your original text? had you said "these new programming hipsters" i wouldn't have argued with you. the truth is that everyone nowadays can program, which is why people get so defensive about that "discipline"
but you guys completely misunderstand the meaning behind the word hacking. so i had to correct it.
yeah, no, try again
get outside your bubble
> The accidents occurred when the high-power electron beam was activated instead of the intended low power beam, and without the beam spreader plate rotated into place. Previous models had hardware interlocks in place to prevent this, but Therac-25 had removed them, depending instead on software interlocks for safety.
I am just a self-taught programmer, and certainly not any kind of trained engineer. But when there's something your machine must not be able to do, and it's possible to design it such that it physically cannot do that thing, what possible excuse can there be for not doing so?
https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha...
Over 100 hours, the system's internal clock drifted by a third of a second, which was enough to introduce a significant error into the Patriot missile defense system. They updated the software the next day when the error caused 28 soldiers to lose their lives.
The Patriot was designed to shoot down aircraft, but ballistic missiles fly much faster than an aircraft, and the warhead fusing system they had back then was too slow.
Most Patriots detonated behind the Scuds they were targeting.
As well as, consider that liability should be part of the industry as it is in other industries.
Sadly the drive for profit speaks otherwise.
When I studied the Therac case, there was also a study in the "Killer Robot" - see here : http://www.onlineethics.org/cms/5122.aspx. Well worth a read to understand how software can become dangerous.
Sadly nowadays, a search for Killer Robot turns up all sorts of stories of drones being used to kill people. When I first looked up killer robot a number of years ago, the results were pretty much all about the ethics scenario and not real life people being killed. How society has moved on...
That is undoubtedly true. However, the commonest classes of bugs can be ruled out by languages with more expressive static verification tools—type systems, effect systems, and so on—leaving us with a stronger baseline from which to work.
This is true, however there are languages which allow for more programming errors than others.
I just mentioned C as one possible example.
The goal should be to have programming languages that reduce programming errors, not that make it easier to shot yourself.
Of course this is only a small step towards better software, as whole system design also plays a big role.
> The defect was as follows: a one-byte counter in a testing routine frequently overflowed; if an operator provided manual input to the machine at the precise moment that this counter overflowed, the interlock would fail.[3]
Can you describe how exactly a language other than C would have helped with this?
Unless the programmer has explicitly disabled them, that is.
Crashing is better that corrupt data.
http://en.wikibooks.org/wiki/Ada_Programming/Pragmas/Suppres...
Modula-2, a possible documentation link
http://www.excelsior-usa.com/doc/xds/isom203.html#15
Just to cite two examples.
* Raise an exception? If so, it's up to the programmer to treat it correctly. There are various ways to detect overflows outside Ada; it's still up to the programmer to treat them correctly. Runtime detection of the overflow (in a hopelessly untested firmware of the Therac-25!) would have brought the matter no closer to resolution.
* Abort execution? If so, is the behaviour of the program defined if execution is aborted? Because undefined behaviour from a control system is quite worse.
SPARK Ada allows the use of source-code annotations to denote information flow in a program, and can vastly reduce the chances of these sorts of problems (at the expense of more programmer time).
Standard Ada compilers can do a reasonable job of detecting some problems at compile-time, but SPARK seems to be the favoured method for safety-critical stuff.
It definitely is! However, it still assumes that the exception is safely handled. The flaw in Therac 25's development process was that testing did not reveal this bug. Consequently, the error recovery process could have itself been incorrect and remain uncaught. It's tempting to think that, if error recovery is similar each time, testing it once is enough, but error recovery is as susceptible to being impeded by race conditions as any other part of the code. No uncertainty is removed this way.
It's also worth noting that a value of 0 (to which the counter overflowed) was actually a legitimate value in the Therac-25 program.
If you are going to say we shouldn't use C because of overflow handling, at least suggest a language that actually does it. Otherwise you haven't really made your case.
Chances are, when you travel by train or take a plane, the control system was developed in Ada.
When human lives are at risk, C is becoming less and less an option.
And when the company still goes C, many countries have certifications in place like MISRA, that make C look like Ada with C syntax.
It's less the language than the ecosystem and the infrastructure. It's eminently possible to develop 'C' that's exhaustively tested - I've done it. I am not sure that is true of C++ or Java.
If the crashed program is controlling a device that is directing radiation at a person's body, then crashing is likely to be at least as undesirable as data corruption.
I understand your point about programming language choice, and your comments elsewhere about the undesirability of the C language. But the root causes of the Therac-25 malfunction wasn't technical, they were organisational. Specifically poor code review and code reuse practices. Changing the programming language wouldn't fix them.
In embedded control software of medical systems that are directly affecting the lives of the patients involved crashing is definitely not an option and it is not necessarily better than corrupt data.
What is an option is to very carefully model the permitted states of the machine and then to exhaustively test those states and their transitions to make sure there are no undefined states the machine can be in.
One way to do this is to have a watchdog process (or even an entirely different system) that monitors the state of the machine and that initiates a safe shutdown if any departure of the defined operation is detected.
In general, the more DRY the code, the more mind-space for review / testing.
This can also be said of choosing simpler design, organizing the code in a better manner, or could have been avoided through more comprehensive testing -- all of which were pointed out by the official report.
The fact that the programmers might have had less stuff to worry about and, therefore, saved their concentration for more critical parts of the code, is hardly direct evidence that the bug could not have been introduced. It can equally well be argued that a higher-level language would have attracted programmers with less experience in mission-critical code, who would have failed to grasp even the possibility of bugs more subtle than this one.
It's something that would be considered firmware today, written in assembler.
Trying to be DRY in assembly is a really great way of introducing subtle bugs.
The software was written in assembly language that might require more attention for testing and good design. However the choice of language by itself is not listed as a primary cause in the report.
Wrong on that one. I do have a good background in compiler design.
> C is not just a portable macro assembler.
Given what it offers when compared with safer systems programming languages, it is one according to my own definition. :)
If you look at the reports of serious failures like this once (and other famous ones), none of them are related to the use of C.
Game: User goes into epileptic seizure, hits their head and dies.
Social media: User calls for help with it - "Please send help #carcrash #bleeding" and the message is lost.
As an engineer and a responsible human being, you don't ever want this to happen. But it will. You have to minimize the chances of it happening, with the realization that it will never go to 0.0%
The world runs on software today. With all that that implies.
Never underestimate the value of a good safety interlock.
This is not the correct way to check for integer overflows: if x + 1 > INT_MAX { ... }
I found it quite interesting.
This page introduced me to Nancy Leveson's work a few years back (her paper on the Therac is linked at the bottom of the Wiki page). She's written two excellent books (and a number of papers) on engineering, safety and complex systems which are well worth reading.