Fatal Dose – Radiation Deaths linked to AECL Computer Errors (1994)
ccnr.org
ccnr.org
> [...] the Therac-25 typically issued up to four error messages a day. It did so by displaying "Malfunction" plus a number, from 1 through 64. No explanation was offered by the computer nor was there any reference to the malfunction codes in the operator's manual. Technicians could, in most cases, bypass the irritating malfunctions simply by pressing the "p" key, for "proceed." Doing so became a matter of habit.
Even if you're not doing safety-critical systems, there are many lessons to be learned just from this bit alone.
Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user.
If you're going to allow users to bypass an error, make it deliberate. The best way to do this is debatable, but through years of bad software, users have been trained to dismiss popup messages by pressing OK or 'yes' or 'proceed'.
When you show an important error, make sure it's different from informational messages or warnings. Showing users routine warnings and messages is fatiguing and basically just trains them to always press the 'p' key.
> According to a computer system's analysis of FDA documents, the computer would not accept new information on a particular phase of treatment (in the case of both Tyler accidents, changing the x-ray mode to electron mode) if the technician made the changes within eight seconds after reaching the end of the prescription data. That's what Malfunction 54 meant.
Consider if they had actually tried to show an error message for this (or even document what it meant). Something like "Data on screen does not reflect actual parameters being used. Proceed?" Hopefully at some point during development, someone would question that: "why are we allowing them to proceed in this case?" "If we can detect that is the case, why even show this as an error, instead of just updating the parameters?"
> Use human-readable error messages, and make sure they are understandable to the users/operators of your software (not just developers). Including a specific error code is fine -- it can disambiguate similar errors, and make it very easy to do a 'find all' in source to go directly to the offending spot -- but that doesn't help your user.
Yes. Remember, however, that much software needs to be localized, even if the initial developer didn't think to. Error codes are much better than hard-coded English-language error messages; map them to a message at the last possible moment.
There are two types of reactions to the Therac-25 story.
1) We need to put a lot of process in place to review and test and ensure we have proper interlocks in place.
2) They just didn't have enough talent on the team. It couldn't happen here.
You want some people with reaction 1 working on a safety critical system.
Ethics and responsible software development was a good unit.
3) I'd never work on life critical medical (or anything else) software, I know I fuck up and even the best can (and I'm not).
I simply wouldn't want it on my conscience that an easily preventable error (in hindsight) hurt someone.
For example, imagine you work in a bank on their mobile apps. You decide that implementing automatic bill payments for a certain type of account is too much technical effort and not worth it.
Hundreds of thousands of people with that account now have a subpar experience. They have to manually pay bills, and many will forget, leading to arrears, bankruptcy, payday loans, and prison.
Your decision not to implement that feature to make the lives of your customers better indirectly led to ruining the lives of tens or hundreds of people, who otherwise would have led a good life.
From an ethical point of view, unknowingly ruining a life due to an action or due to inaction, is pretty similar.
https://en.wikipedia.org/wiki/MIM-104_Patriot#Failure_at_Dha...
> The Patriot missile battery at Dhahran had been in operation for 100 hours, by which time the system's internal clock had drifted by one-third of a second. Due to the missile's speed this was equivalent to a miss distance of 600 meters.
TSA employees are not permitted to wear dosimeters[1], which makes it hard to detect these things. Scientific American, the most established magazine in the US, is among respected bodies that have recommended dosimeters for accountability reasons.[2]
There are also extremely infrequent (and sometimes rigged) studies done on the equipment, and they often find dangerous conditions. For example:
>Doses for some of the baggage screeners exceeded the maximum dose for the public.... Some EDS machines were not well maintained (i.e., they had bent curtain rods and missing curtain flaps).... Most EDS machines emitted low levels of radiation; a few exceeded regulatory limits.[2]
[1]https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-... [2]https://www.scientificamerican.com/article/airport-screeners... [3]https://www.cdc.gov/niosh/hhe/reports/pdfs/2003-0206-3067.pd...
Zero. The TSA machines are not limited by software. They will catch fire themselves before killing people. Therac-25 was designed to destroy tissues, cancer tissues but human tissue nevertheless. It is like comparing an m-16 to a rubber band gun. They are both guns, but they are fundamentally different in terms of capabilities with the former.
At the airports. They still exist in prisons, in covert trucks and other mobile units owned by TSA and law enforcement, and potentially in other venues of public transportation, stadiums and arenas, etc.
X-rays are also in INCREASING use at airports, since they are in the process now of replacing the carry-on bag scanners with CT scanners. A CT scan gives a 3D image of your luggage which means you don't have to take out your laptop. Unfortunately, a CT scan also uses 10X more radiation than a regular x-ray.
1. Xray scanning should be outlawed for public safety use in humans. You want to scan humans? You must use millimeter wave, and the cost is on you.
2. Governance. It's perfectly acceptable to use CT scanners for non-human surveillance (carry on bags, checked luggage, etc), but you must ensure life safety is enforced. If you don't, steep penalties up to and including jail time.
Call your reps, run for office, vote for people who care about this.
When I was still in VT I was happy to see that our third tier airport managed to hang on the noticeably more effective metal detectors long after other areas phased them out, sadly they've now been swapped out.
https://www.cbc.ca/news/canada/hamilton/radiation-treatment-...
http://indico.ictp.it/event/a12208/session/65/contribution/4...
PRetty damning, if you ask me.
Here's their fationale from teh FAQ in the linked article [1]
"Q: Why aren’t your officers permitted to wear dosimeters?
A: There is a really good reason for this. The emissions from our X-ray technology are well below the requirements that would require their routine usage. To help reassure passengers and employees that the technology is safe, however,health physicists with the U.S. Army have been conducting area dosimeter surveys at multiple airports nationwide."
This is at best massively flawed reasoning by the TSA, and at worst a deliberate misleading rationale.
1) While the NORMAL dosage is of course well below any level that would make a dosimeter useful, a dose from a malfunction or bent/broken/missing shield panel could easily exceed dosimeter detection thresholds.
2) Spot checks are nice, but unless they are very frequent, they could easily miss a malfunction and leave it overdosing agents and people for a long time. Even if inspections are every 6 months at every airport station, that will leave a malfunction overdosing people for on average, three months.
3) Detecting, identifying, and repairing a malfunctioning system could be done much sooner and at lower cost by using dosimeters, either on employee volunteers or just hanging them on the chairs & workstation surrounds.
It is this kind of frankly stupid stick-your-head-in-the-sand reasoning that breeds distrust of governments and the companies that supply them.
[1] https://www.tsa.gov/blog/2011/06/30/tsa-cancer-cluster-myth-...
That doesn't mean they aren't allowed to perform dosimeter readings at all, it just means it's not acceptable to carry this equipment around all the time.
> While the NORMAL dosage is of course well below any level that would make a dosimeter useful, a dose from a malfunction or bent/broken/missing shield panel could easily exceed dosimeter detection thresholds.
Do you actually know that these could be harmful, or is it just a suspicion?
So I'd assume this is more like coal miner operators in the 1800s planning ahead to protect themselves from the eventual outbreak of black lung.
As far as we know, there is no general ban on wearing instrumentation of any other on the job, unless it is somehow clumsy and would interfere with their duties or the appearance of their uniform.
In contrast, the simplest dosimeter is typically a very small patch of photographic paper in an envelope opaque to visual light but transparent to harder radiation (towards the gamma end of the spectrum). It is like an inch square, or like a typical nametag, easily worn in a shirt pocket, no more visible to anyone else than a credit card in the shirt pocket.
The only plausible reason to forbid this is to prevent any kind of whistleblower event.
Of course I do not know the actual levels of radiation inside each machine or all of their possible failure modes. However, we do know
1) high-energy radiation dosage is cumlulative -- each day's dose adds up, so if you have 1/100th of a critical dose each day in a workzone, in less than four months, you should be banned from that work zone. This is the way it works in many facilities dealing with radiation sources, and everyone wears dosimeters which are checked, and people are reassigned when the total dosage exceeds the prescribed level.
2) There are medical limits on how many x-ray and CT-Scan procedures are allowed for patients, again due to the cumulative nature of radiation exposure. Even with the very low-doses of modern machines, these are a concern and are tracked (I know this from the experiences of several close family members).
3) The TSA machines do produce sufficient radiation in their beam to fog ASA800 and higher film. This is definitely a non-trivial dose. Tho obviously not lethal in a single dose (e.g., if you rode through the machine), it is significant. If an interior shielding panel is bent or somehow mis-installed during manufacturing, maintenance, or repair, it is entirely plausible that people working regularly in the area of the leaking radiation could be overexposed.
So, no, this is not even close to mere suspicion or Luddite "ooh radiation bad" -- and yes, we do know that these could be harmful, especially over extended periods.
I hope you are not an employer with that attitude.
Who is "we"? Aren't you just pleading ignorance here? What do really you know about what TSA agents can and can't wear? Anything can "interfere" with the appearance of a uniform, including a "small photographic patch".
> The only plausible reason to forbid this is to prevent any kind of whistleblower event.
Then why do they do allow third party monitoring?
Having further looked into this, the page you posted is outdated and they ended up handing out dosimeters themselves:
https://www.livescience.com/36079-airport-security-screening...
The result was negative. Of course you can't prove a negative, maybe there's radiation somewhere else where they didn't look...
> 2) There are medical limits on how many x-ray and CT-Scan procedures are allowed for patients, again due to the cumulative nature of radiation exposure.
> 3) The TSA machines do produce sufficient radiation in their beam to fog ASA800 and higher film. This is definitely a non-trivial dose.
According to: https://www.radiologybusiness.com/topics/business-intelligen...
" TSA claims the “backscatter machines” cannot produce more than 0.005 millirem of radiation per scan. In comparison, the agency says, a chest x-ray exposes patients to 10 millirem of radiation, and the maximum recommended exposure to radiation from man-made sources is 100 millirem per year."
That would mean that if you had a machine that was completely unshielded, you would have to perform 2000 scans to get the equivalent of a chest X-Ray. Compare that to the radiation exposure of your average flight attendant.
Not if it is inside your pocket. Get real.
>> Then why do they do allow third party monitoring?
Obviously because they can control when it is done as well as how the results are distributed, or not distributed.
>> " TSA claims the “backscatter machines” cannot produce more than 0.005 millirem of radiation per scan. In comparison, the agency says, a chest x-ray exposes patients to 10 millirem of radiation, and the maximum recommended exposure to radiation from man-made sources is 100 millirem per year."
>> That would mean that if you had a machine that was completely unshielded, you would have to perform 2000 scans to get the equivalent of a chest X-Ray
1) this assumes that the "0.005 millirem per scan" is the total output of the machine, including all radiation absorbed by the lensing & shielding. We do not know if this figure is the total generation output, or only the total amount put to the target & receiving sensors.
2) one thing we do know is that the radiation will NOT be scattered evenly; there will be lobes and hotspots and cold spots.
3) even if we assume .005millirem/scan, at the rate of two/minute, that's 1000 minutes to get a chest x-ray, or 16 hours, basically two work days. In LESS THAN ONE MONTH, they are over the yearly dose.
More importantly, this is only counting the backscatter machines that are specifically designed for super-low doses as they are specifically irradating people.
This does NOT look at the luggage scanners, either for carry-ons or for checked luggage, which likely have much higher outputs.
Finally, even if your arguments are correct and there is truly no risk, there should be no problem at all for officers to wear their own dosimeters -- the results would merely be putting their minds at ease that they are not getting exposed.
Thus, we are back to the only reason to prevent officers from wearing dosimeters is an attempt to control the information and prevent whistleblowers and/or lawsuits.
Again, I sincerely hope you are not an employer with this attitude.
Let's apply some Occam's Razor here.
I am led to believe they know that these machines are harmful and therefore they are carefully orchestrating all testing of these devices by, among others, all of the following parties: "U.S. Army Public Health Command, the Food and Drug Administration’s (FDA) Center for Devices and Radiological Health (CDRH), the National Institute of Standards and Technology (NIST), and the Johns Hopkins University Applied Physics Laboratory (APL)".
That's all done to prevent some sort of mass lawsuit, which might cost the United States some money.
The simpler explanation is that TSA rules are simply really strict, because that's just how TSA rules are.
https://www.quora.com/Are-TSA-screeners-permitted-to-wear-ra...
The rules don't have to make sense. There's a whole lot about the TSA and its procedures that doesn't make sense.
> We do not know if this figure is the total generation output, or only the total amount put to the target & receiving sensors.
I'm content with the description given that these machines "cannot produce" more than that. You can do your own research on the matter.
> 3) even if we assume .005millirem/scan, at the rate of two/minute, that's 1000 minutes to get a chest x-ray, or 16 hours, basically two work days. In LESS THAN ONE MONTH, they are over the yearly dose.
...assuming no shielding whatsoever. You would hope that after two months without shielding on the device, someone might notice. Even then, such exposure would be well below the occupational limit for radiation workers (5000 millirem):
https://ehs.stanford.edu/manual/radiation-protection-guidanc...
> This does NOT look at the luggage scanners, either for carry-ons or for checked luggage, which likely have much higher outputs.
Fair enough, but you picked an article that was specifically concerned with those backscatter devices.
That's not even close to the simplest explanation. Far simpler is that they are careless, clueless, and paranoid, and don't want anyone finding out something bad on their watch. Or they just don't want to allow anyone to get concerned.
>> The rules don't have to make sense. Unless we're living in a Kafka or Vonnegut book, they are supposed to have actual reasons and make sense.
Calling out one particular type of measuring device, which happens to measure the one plausible type of hazard in that workplace, yeah, that's suspicious.
>> .assuming no shielding whatsoever. Does not assume no shielding whatsoever, only a plausible hotspot or focus. The point is that the potential dosing is in the range of values that could cause a hazard, and even at 10% of that value, would be overdosing within a year. It is not like it is totally off the scale (as are many of the public's fears).
>> You would hope that after two months without shielding on the device, someone might notice.
Yes, you would hope that. However, if you've ever read reports on industrial issues, you'll find that far worse and more noticeable hazards go unnoticed/unfixed for YEARS at a time before they get around to killing someone. And this sort of low-level radiation is particularly pernicious, as it is completely unnoticeable until after it is too late.
The simple solution is, assuming that these machines are largely safe (which I generally expect), is to say "you can buy dosimeters on your own dime and wear them if you like, just keep them out of sight in your pocket while on duty. If anyone gets any unexpectedly high readings, we'll investigate, but remember since you haven't got a chain of custody around the meter, it'll be only the beginning of an investigation, not immediate acceptance of your results."
Ye, it may turn out to be bit more work, but it shows you're on the side of worker safety, and on the road to getting things fixed. Mostly, it'll quell any fear and likely cause no one to wear one.
With the rules as it is, I'd be damn sure to sneak dosimeters into my clothes while working there.
Thank God for the curious minds of this world, the people who have he patience and dedication to reproduce bugs.
Sorry for linking the daily fail, but the only articles I can find in English seem to be in UK tabloids: https://www.dailymail.co.uk/news/article-7066289/Cancer-pati...
по-русски: https://dni.ru/regions/2019/5/23/424405.html
As an aside, every time I step on the treadmill at the gym I check that the emergency stop button works and that the incline down and speed down buttons work. I have been stuck on a treadmill which I couldn't stop or slow down before.
Mechanical failures need to be dealt with differently than software. Parts must be replaced before they fail. Even if they appear to work correctly. Software doesn't wear out so its a whole different ball game.
Sounds like there is an obvious design failure - similar to garage or elevator doors the table shouldn't move and push up beyond some force threshold (patient weight plus say 200N of force).
200 errors I think.
Look at the rest of the document. It is almost clear from the errors you can find throughout the document that this "report" was posted from some original hard copy run through some form of OCR.
I think this is a particular example of it. Why the number of LOC is 101000?
Maybe the OCR interpreted a bad comma from scanning as a "1", in which case the number would have read 10,000 LOC - and thus "twenty errors".
As another glaring example of this OCR issue, look for this line:
> "There were [.4ECL] people sitting in our offices telling us it [the Therac-25] couldn't hurt anybody when they knew it could."
"[.4ECL]"?
It should read "[AECL]" - but the scan or the copy it was scanned from probably had a small break on the arm of the "A" - which the OCR interpreted as ".4"
You can find other small anomalies like this elsewhere in the document if you look for them. They kinda bugged me, but I could tell from the pixels what was going on.
;)
EDIT: My own mistakes fixed...sigh
Design the security/safety system up front, for the task of the system being designed, for all potential failure modes - and then implement the first, and test the hell out of it, and document it thoroughly.
Unfortunately, this tends to get shot down, until it is needed, and then it gets bandaid-patched in place, because usually at the point it is needed, a whole refactor of the code base is usually called for, and we can't have that, so "just fix it" becomes the call of duty.
And it never works out properly. Things get missed, security is effectively out the window, until something happens, with the refrain of "why did this occur" and of course nobody wants to take the blame for "I told you so".
Unfortunately, even implementing such a security system first will only get you so far, and it is still possible that something might be missed, and potentially be the cause of a safety critical issue. But hopefully, most if not all of those extreme events can be caught first in the system, by some part of it, and what's left will be relatively minor.
What does that actually mean? Maybe in the Therac-25 case the maximum dosage could somehow have been limited by a hardware dial in addition to software. But what form would hardware safeguards take for the millions of decisions software makes every second on a starting rocket? And considering hardware has far higher failure rates than no-moving-parts software, wouldn't any system that lets hardware dominate software decrease safety?
From Wikipedia:
"One, when the operator incorrectly selected X-ray mode before quickly changing to electron mode, which allowed the electron beam to be set for X-ray mode without the X-ray target being in place. A second fault allowed the electron beam to activate during field-light mode, during which no beam scanner was active or target was in place.
Previous models had hardware interlocks to prevent such faults, but the Therac-25 had removed them, depending instead on software checks for safety."
A dose integrator would be a detector + an integrator independent of the therapeutic delivery software. If it goes over limit, it shuts the system down.
Discussion of another good article on the Therac-25 (with pictures of the machine) [2]
Astonishing this basic safety precaution was not already in use. Do modern devices do this?
I guess nobody thought about Murphy's Law.
> there have been complaints in the high-tech community that software documentation is hampering competitiveness
The previous title noted that this was bout the Therac-25.
...and I have to add that I'm one of those "informally educated software engineers" - though I've never touched or worked on any kind of safety-critical systems (and I would decline such a position if offered, to be honest). I've been doing SWE for over 25 years now, and the only "formal schooling" I've had in the field was a couple of community college classes I took to learn C/C++ 2+ decades ago. Everything else has either been learned thru my employment, or at home. Prior to my first "professional" position in 1992, all I had was a high school degree and an associates from a trade school in "computer electronics".
But even I know about the Therac-25, and it's always in the back of my head.
The fairly recent change in the industry surrounding automated testing, test case programming, CI, etc - all of that which more or less fell out of the whole "agile" movement - has led to a vast improvement in software, imho. Yet bugs still persist, despite all of that and more.
We have to do better - I don't know how, or if it's even possible (from what I understand, it's not mathematically possible, except in maybe certain trivial examples, to validate all possible states of code - maybe I'm mistaken on that).
It's good when not every title is completely obvious. It makes readers work a little and gets the brain out of internet reflex mode.
https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
Also, there's a historical aspect to this. We don't need, and needn't presume, to rewrite the titles of classic articles.
And yes, this is art not science: aimed at effect.
The Therac case study has acquired a recognition the contemporaneous article (and title) wouldn't have experienced -- the company name was then the more notable signifier. AECL has been far eclipsed by its most notorious product. Today, "Therac-25" should lead, on the same basis as "AECL" did in the original.
I'm generally a follower of Jacob Nielsen on microcontent:
Well-written, short text fragments presented out of supporting context can provide valuable information and nudge web users toward a desired action.
https://www.nngroup.com/articles/microcontent-how-to-write-h...
Mrtimer J. Adler does not fully condemn content concealment in book tables of contents, but his lips are clearly pursed, and nose wrinkled:
It used to be a common practice, especially in expository works ... to write very full tables of contents, with the chapters or parts broken down into many subtitles indicative of the topics covered. Milton, for example, wrote more or less lengthy headings, or "Arguments," as he called them, for each book of Paradise Lost. Gibbon published his Decline and Fall of the Roman Empire with an extensive analytical table of contents for each chapter. Such summaries are no longer common.... [P]ublishers have come to feel that a less revealing table of contents is more seductive than a completely frank and open one. Readers, they feel, will be attracted to a book with more or less mysterious chapter titles-they will want to read the book to find out what the chapters are about. Even so, a table of contents can be valuable.
-- How to Read a Book, p. 33.
The same argument applies to article titles at HN.
You've made the case (and commented on comunity failings) for defusing titles on hot-button subjects. That's valid.
This isn't that circumstance.
Pandering and information concealment as a deliberate policy, again not the case for the article as initially written, in the cotext of its time, is overtly manipulative, to no real gain.
Please don't do that.