How a robot pharmacist wound up giving an overdose
medium.com
medium.com
The most common problem is when the people setting alert policies are divorced from the people receiving them. The managers have no direct incentive to keep the alerts sane or moderated. On an individual basis, every alert that exists shifts potential blame from the manager to the employee. ("It's all the employee's fault for not following the alerts.")
In this way, over-alerting is often "automated micromanagement", but without even the feedback-loop and mutual-accountability of actual micromanagement. Until the people acting on alerts have some feedback into the alert-configuring process, the problem will continue.
Another dysfunction is when people ask for alerts purely for an independent paper trail, either because they're paranoid about assigning blame, or because they want to use their e-mail client as a half-assed searchable database.
P.S.: We've already had managers asking us if we could have e-mail alerts that users must confirm receipt of by visiting the website... and for the site to continuously e-mail followups until they do. Meanwhile, the IT staff have their own horror-stories of ancient inboxes grown too large to reliably read data from.
I'm sympathetic to "it's corporate's fault", but instead of fixing that problem I see a lot of folks lobbying for shiny new toys, meetings to stroke egos, and basically spending their political capital on being prima donnas instead of making their lives easier from an operations standpoint--and for the rest of the docs such as yourself this obviously is a problem. :(
The roadblocks bit is what it is--if doctors were better at prescribing medications and hadn't cowed their colleagues and nurses into a culture of "oh, well, I shouldn't say anything" then automated systems wouldn't be so compelling from a safety standpoint. As it is, they are undermined continually (as explained in this article) by the humans who have to work with them and who--as you've exemplified--don't trust them.
Let's face it: in the next 30 years, most physicians who are basically knowledge workers are going to be doing work very differently than they are today, most of it with heavy computerized assistance. You can choose which side of history you want to be on.
I remember when my mom had radiation treatments for brain cancer, which were so awful they killed her quicker than the disease would have (by a few weeks). A few months later I read about how terrible the design for calibrating these machines was leading to a large number of mis-treatments, my sister and I wondered what dose she actually received.
During the Vietnam war pilots turned off all the alarms in some of their planes due to constant alarms that always happened during intense combat situations leading to an inability of focus on what mattered.
UX design is not easy but has to be treated as more important than just making it pretty. Understanding how real people use these things is incredibly important and not just a last minute bullet check.
Amen brother.
The last four large scale business enterprise applications I've worked on, design was left to either the back-end developers, project managers or done by committee. The ones done by committee were absolutely brutal. Nothing rips your soul out quicker than trying to get 20 or more people to agree on something.
I have no idea when or where design just got completely buried and left to people who clearly have no idea what they're doing, but in your case, it's incredibly troubling and scary.
Most programmers I meet don't even understand the importance of it looking nice. They think that the user should do their due diligence. :(
There was Duke Cunningham[1], a fighter pilot ace that (IIRC from a documentary) also turned off the comm link from his RIO to avoid the potential distraction. He was able to speak to his RIO, he just wasn't interested in what the RIO had to say to him. Wow.
His political career later went down in spectacular flames.
In my opinion, this is a dangerous misperception. Hindsight is 20/20. You saw one screenshot displaying the incorrect dosage; the doctors see "scores" each day, day in, day out. The brain is hardwired to ignore repetitive stimuli. Humans are smarter than computers overall but have a much higher error rate; "competence" can decrease that rate, but not to zero. There is always a chance of a mistake.
The software could have been designed to make the probability of requesting a 38.5x overdose from the robot pharmacist negligible, by reserving a special warning for extreme circumstances. This requires the programmers to do a non-shit (shall I say "competent"?) job once, but not constant vigilance by the doctor, who I'm sure can use the saved mental cycles to check for many other potential errors the machine can't handle completely trivially.
I expect a more thorough design overhaul would cause a non-negligible reduction in the occurrence of a wide variety of mistakes, dangerous and not.
Recommended dosage
|
----|-x------------
----|------------x-
A special warning is good. But the software could ALSO require the doctor to, instead of just clicking, read a message like this on the screen:
YES I REALLY WANT A DOSE OF 6160 MG OF (whatever)
There is nobody, no matter how busy, who would type those words without thinking about what they are doing.
Of course, that sort of typing can't be requested routinely, only in extreme circumstances. Perhaps these extreme circumstances would require TWO people to type that same message?
Or, maybe the software should simply REJECT a dosage greater than perhaps 10x safe dosage for a person. No override allowed?
That here is the problem, right here. If we can have self-driving cars (which can look ahead and figure out the result of their actions), so can a system like EPIC. Why can't it calculate the probability of a Bad Thing(tm) happening, and suitable tag the alerts?
It almost certainly put 38.5 pills per day in a bottle. At which point the dispensing Pharmacist or Assistant should have started asking questions.
Doctors -up these kind of things all the time; around 5% of prescriptions based on the very scientific measurement of the relative size of the good/bad stacks on my wife's desk.
One other thing that went wrong here - through no fault of her own, the mom was in a different room with the brother. She knows what the patient normally takes, and would likely have started asking questions when she saw her son taking way more than what he usually took.
First, Epic does a lot of custom work--as noted in the article, each installation is "customized" to only have particular alerts. This makes it hard to count on a standard system behavior, because there isn't one. This continued coddling of clients makes it very hard to introduce new software into the market, because you can't do it without deep pockets and sloooooow sales cycles.
Second, the physicians are very much overloaded with alarms and alerts--and instead of saying "Hey, let's figure out why we keep ending up with so many alarms" they instead do the short-term thing and just ignore or turn them off. The entire profession is rooted in split-second decisionmaking, for reasons I cannot fathom. They really, really, really want to be the heros that come in and save the day, and systematic approaches to operations optimization just aren't a priority.
Third, the data coming off of these things (and at work we've seen this) usually indicates large operational deficiencies. Unfortunately, you can't just tell the clients "Hey, you all are doing things wrong, and here's the proof", because they either won't care, or they will fire you and bring in somebody who keeps them happy, or they'll care but be unable to make changes because that introduces uncertainty into their line of work and that could hurt someone--nevermind all the existing casualties.
Because insurance companies, Medicare, and Medicaid reimbursements barely cover costs, so to be profitable they have to make it up on volume.
If you compare the operators of decades ago with the sysadmins of the 90s and the devops folks of today you can see a rough path of evolution that hopefully medicine will follow: a last-minute heroic effort to save a system is a failure, not a victory.
Right. There are very few points in time where decisions in medicine need be like this. Even in the ED, where I interact most with physicians. Hell, the ED I see most often has checklists to "stop and review" (and they do) for things that most people would consider (and are) time-critical, cardiac arrests, intubation, and the like. It's odd to get used to, but seems to work very well.
Did this used to be true? I have never encountered this. Holding down Shift, perhaps, but not caps lock...
http://ux.stackexchange.com/questions/64230/why-does-the-cap...
Not sure about the Caps Lock thing though.
You know, given what we know about the liver toxicity of acetaminophen/paracetamol at this point, not sure this is the best example of a spurious alert. There's a good argument that this is one of the more dangerous drugs OTC, in part due to the widespread impression that it's not very dangerous.
It is interesting looking at the story in general as an example of alert fatigue, though, and the need to differentiate different types of alerts.
Unlike some of the other comments I see here, I don't think this points to incompetence so much as human nature. It takes a fair amount of cognitive power to overcome the natural tendency to assume that when the alert was spurious yesterday, it's not spurious today.
That power would be better spent elsewhere, and points to a failure of automation to truly lighten the load.
"Hey, you're almost taking exactly the amount of this thing we say you can take. Watch out!" is pretty much the definition of a spurious alert to me. It's telling you that what you expect to happen is happening.
But either way, in this case I'm pretty sure Tylenol was stressed in the article because it was an example of an innocuous medication, which it really kind of isn't.
Why is acetaminophen even in the picture? Isn't ibuprofen very similar in effect but much safer in an overdose?
Can't speak for why it'd be favored in children, but NSAIDs do conflict with a number of medicines used in adults.
Terrifying example of one way AI or computer-augmented decisions can go horribly wrong. A "Mode error" is at the core of the difference between intelligence and common sense.
This is similar to the "Trolley Problem" ethical thought experiment—we have a computerized system, that will save a lot of peoples' lives—at the cost of others, who wouldn't have otherwise died. It's a hard problem, with no easy solution. See: http://en.wikipedia.org/wiki/Trolley_problem
The UI screenshot in the article makes that mistake VERY clear, and their "are you 100% sure" question is obviously intended to illicit a particular response (who is ever really 100% sure of anything?).
I know this article is about how systems can be altered to protect patients, but much like with the NTSB's findings, sometimes it legitimately is pilot error (or doctor error in this case).
Can things be better? Yes. Absolutely. But so can doctors. These types of systems are stop-gaps, without them this doctor could have calculated the wrong dose by hand and made just of large of a mistake.
When you routinely get prompted with that question dozens (hundreds?) of times a day, it can't elicit that response successfully. It's just not how our psychology works.
The doctor didn't order 38.5 tablets -- at least she didn't think she was. She thought she was ordering a dose per kg, not a quantity of tablets.
You say the screen is very clear. It's clear enough when I read it. I had a good night's sleep and I am alert and awake. I've been primed by the article to look for the error. I imagine I would easily miss the mistake if I've been working hospital wards for the last 10 hours on a rolling nightshift, looking after some very ill patients.
The fact that it's possible to use two entirely different dosing systems to order medication and have the screen not clearly indicate[1] which is being used is so obviously sub-optimal I'm kind of surprised to see anyone saying that it's "just pilot error".
Actually, she thought she was ordering a single dose at 160mg, and accidentally ordered dose per kg. 160 was the size of the dose, so the patient's weight became the number of pills.
If the doctor is used to working with older/larger patients (over 40kg) they might get that screen only rarely, and perhaps because the patient was so close to 40kg, she wasn't thinking in "pediatrics" mode, and expecting it to come up.
Seriously, what would that cost, like $20? (Obviously there are a lot more things wrong with this system than just using a mouse for too much, but this would help). We should be willing to invest more in our machines: http://www.loper-os.org/?p=861
My wife is a Pharm.D. at a hospital (that coincidentally is currently installing Epic). I know she worries (obsesses even) over the possibility of making an error that hurts or kills someone. I'd like to think other people doing that work feel the same way. Even if she can get three or four nines worth of confidence in every decision she makes however, that leads to some unfortunate probabilities when multiplied out by a career's worth of decisions. We can improve the technology to reduce the chances of significantly negative outcomes but they will always be non-zero.
I don't envy her or the programmers working on systems like Epic. It's messy work with serious consequences.
If we can filter out the Nigerian prince scams from your email, why not apply that to save lives?
If these people in the course of their work choose to ignore alerts, are distracted and not actually doing what they were hired to do (hint: theres a reason there were three humans in the chain here), the fault lies on them, not on the software.
You obviously can't have it both ways. We call it automation because things will happen automatically. At the end of the day, this system probably causes significantly less problems than the old way of trying to read some doctors handwriting only to pick up the wrong medicine regardless.
>Every training program has a “hidden curriculum” (the way things are actually done around here, as opposed to what the policies say or what the administrators told you during that interminable orientation). One of them — passed down from senior residents to the newbies — was, “Ignore all the alerts.”
I don't disagree that this shouldn't be considered carefully, but it's a little harder than maybe you're making it seem.
But if you think /any/ information is easy to see in that rat's next of anti-design, I think you're wrong. I
So clearly, these people simply didn't look. Not once. In their perpetual multitasking frenzy that any hospital staff seems to be in, they didn't do the actual task. I'm reasonably sure that you can color code or flash whatever yo u want and they would still miss it if that's the way they work.
I still don't understand how the nurse administering the dose would not have been alarmed that the kid was apparently supposed to swallow almost 40 pills at once. That's the sort of unusual situation you WOULD expect a human to notice right away.
What you can not do responsibly is to simply ignore all of them and then claim that somehow it's the fault of the software. Yes, as it turns out, you actually take on responsibility when you work in these kind of areas. I personally would never agree to code for systems that end up radiating humans or other safety critical systems.
Who do you think buys the equipment or has contact with the vendor?
This is where your mistake is coming from. You think for some reason that people are choosing to ignore the alerts, rather than being habituated to ignore them.
It is theoretically possible for a person with a decade or so of training to maintain a constant level of attention on repetitive trivia. Studies on Buddhist monks have shown this. But the people who were doing this didn't study Buddhist meditation, they studied medicine, so expecting them to behave like Buddhist monks, which you seem to be doing, is the height of irrationality.
Humans have very limited attentional resources and extremely powerful internal non-conscious filters to conserve those resources that can only be overcome by lengthy and rigorous training, which would exclude the kind of training these people actually needed to do the rest of their job. These are just facts, well-established by a wealth of empirical research over the past several decades.
Responding "Give me a break" to a system whose failure was fully predictable based on known facts of human psychology and physiology is precisely the problem. Perhaps someone on the design team pointed out the issues, and were told by someone who knew nothing about human psychology beyond what their gut told them, "Give me a break!"