Error by AI scribe during medical appointment leaves patient devastated
abc.net.au
abc.net.au
If it turned out an LLM embezzled funds and spent them at an internet casino, we would have folks in the comments explaining that what really matters is the embezzlement rate compared to humans doing the same job.
In addition to not enjoying my work as much as what I used to because it's become babysitting an superpowered AI toddler, I now have to deal with this kind of opinion online.
That's hopefully temporary: you are dealing with provisional architectures - as is obvious by their lackings (transparency; reflection; evolution; one shot learning...).
The very fact that you use the term 'AI' for LLMs when some of us would not ("NNs are used in AI" does not mean that all NNs would be AI), or would be wary of that use signifies a problem that is being tackled and will be worked on until the next stage.
Because thats how probabilistic machines work. You can’t change that.
Clearly AI has been extremely useful for doctors despite its “flaws”.
I dunno, for every new hype automation, the machine is given a lot more leeway then people. At least by some on HN. I remember FSD discussion years ago where FSD was already supposedly better then people and all its problems explained away.
I think this sums up what I find most frustrating about all of this from the executive level. It feels like everything that was important just a few years ago is now considered unnecessary baggage, that is merely there to slow everything down. What would've got you sacked is now applauded at times, and this is only really a year or two into it proper.
Based on watching the medical software field as a consumer (patient) and friends who are doctors, this is a fantasy. The quality of software in this field is abysmal and there seems to be almost no repercussions to those who develop or sell it.
Which is precisely why this sort of thing can be rolled out without much fear by those pushing it.
I get why it's problematic, obviously, but if it produces statistically better results (which I have no idea of), I don't think it's right to just write it off because of this.
I do not oppose AI integration; I'm not a Luddite. But having a "move fast, who cares if a couple die" isn't the way to go with sensitive fields, like the medical field.
I suppose we will come up with proper responsibility-hierarchies and guardrails around AI, but until then, people have a right to complain about the lack of them.
Automated transcription for anything official is scary to begin with, because some noise in the background is all it takes to turn "I've never taken mushrooms" to "I take mushrooms," or whatever. And then the LLM will simply report "Patient reported using mushrooms."
One of the main issues is around homophones in an accent (Adam/Atom in American English, Bath/Barf in London English, etc.). Not to mention pronunciation variations due to fast speech, speech impedements, or parts of words side-by-side that sound like a different word.
Another big issue is around misaligned training data. For example, Whisper is known to hallucinate on silence [1].
[1] Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio (https://arxiv.org/html/2501.11378v1)
It should be completely and utterly intolerable that a computer produces a different output given the same input. We shouldn’t couch that behavior in soft terms like “hallucination”. A computer system that non-deterministically makes mistakes is a defective computer system.
1. accents -- Especially around mergers (cot-caught [AmE], trap-bath [BrE] vs palm-bath [LondonE], pin-pen [Some AmE]). These can even be hard for native speakers -- try transcribing a broad Scottish, London, Brooklyn, or Indian accent and see how well you do.
2. sound/phoneme variation based on surrounding phonemes -- It is common for the 'n' sound to be realised as an 'ng' sound before a 'k' or 'g' sound due to velarization ('ng' is the velar variant of 'n' and 'k' and 'g' are velar sounds). It is common for vowels to be nasalized before nasal sounds ('n', 'm', 'ng'). It is also common in non-rhotic (don't pronounce the 'r's next to vowels like in 'start' and 'north') to pronounce an 'r' between two adjacent vowels in words ending/beginning with vowels (the "intrusive r", e.g. in "there and back").
3. sound changes due to fast speech ("I'm gonna see 'bout it t'day.", etc.)
4. ambiguity about where words start/end (e.g. "to Damon" vs "today mon" where the "mon" is the variant of "man" in Caribbean English).
5. word play, puns, etc. due to accent and other speech.
6. technical words in a given domain, specific place names, etc.
7. other things that can affect speech such as mumbling, stuttering, or slurred speech.
"AI".
If concepts can be that sloppy, then the party that believes it an argument that NNs surpass humans get a point.
Edit: in fact, there is a point: we compare AI (proper AI) to optimal professionals, but that is not the real scene. And this is why in computing we bet on deterministic algorithms: they do not guess a solution, they compute it. There is no comparison with the possibility of failure from a biology based system - in deterministic computing the failure is restricted to exceptions.
https://en.wikipedia.org/wiki/British_Post_Office_scandal https://en.wikipedia.org/wiki/Robodebt_scheme
It seems to me that most people regard computers as some kind of infallible truth machine. If told its spewing garbage they're more likely to double down and shoot the messenger than try and get it sorted out.
My problem is who is accountable when the AI is given autonomy and messes up
It seems like AI is being deployed so it can take the blame for some individuals decisions that will have negative impacts. Then they can shrug and say "wasn't me, it was the AI"
Or even worse they hold a fall person accountable. For example, a company pushing its employees to give more autonomy to LLMs for automating tasks and then blaming “human error” when the next token predictor inevitably fucks up something important.
I can say for a fact that reliable medical transcrption and dictation is worth handling hallicinations..
its an order of magnitude worse in real life.. or else its just ommitted info since most docs and nurses dont have time for details..
By all means lets be accurate but we must remember all these complex workflows are filled with human error..
the problem is that introduction of any new technology usually happens with so much emotional baggage, that when there's an error (human or otherwise) some humans will understandably see their biases confirmed in them, and will signal boost everything to the Moon.
LLM's will never be reliable enough to let loose on tasks that require 100% accuracy, therefore a human will have to review their work. So will any time actually be saved, or at least enough time to justify the cost and extra complexity of the new system?
And I am sure people will still defend that dystopia with "companies send canned response all the time".
Also, the types of mistakes are completely different. A person may mishear something and ask to verify; LLM is always certain that what it transcribes is a fact. A person might omit something but won't make up the facts like that.
So, a mistake is not the same thing as a hallucination.
"the nurse who was trying to be very nice said in a less sweet tone that the doctor is supposed to review the notes so this does not happen. And that she is very glad I caught it and the doctor will be glad too. I do not know if this was something Very Serious or an ongoing problem or what."
Given this threads story is the second incident of this I've heard in a week it seems like it is a common error with very serious consequences.
So even the doctor forgot what they had talked about, and assumed the transcription was right? Crikey.
That's $50 billion a year, and a million people pulled out of the labor force. Is that worth it? Is that really the best thing you can do with $50 billion dollars? In real life, tradeoffs exist.
The humans are probably making mistakes like writing 100mg of something when they meant 10mg, which can also have dangerous consequences, but to an extent it's known this happens and there are processes to catch it. The type of mistakes and how much trouble and distress they cause matters as much as the number.
https://pmc.ncbi.nlm.nih.gov/articles/PMC7284300/
"It was stated in notes that I had lung cancer. I do not and never have had lung cancer."
"I did receive a referral for physical therapy, it was for the wrong body part"
"Doctor reported that I did not claim to have pain in my hand. I am a pianist and I went specifically because pain was in my hand."
"I have been complaining of difficulty breathing [for over 3 mo]... notes saying my breathing is normal"
And plenty more
(I do think this is extraordinarily rare compared to how often AI hallucinations happen, but it's such a bizarre story that I have to mention it.)
From personal experience I can tell you that it does indeed happen. Not specifically micro-dosing psilocybin of course, but of a similar nature, and completely made up. Not sure how it happens.
Damn, my doctor is always happy to give me 45 minutes of her time or more. Sometimes I feel like I'm taking up too much of her time chatting her ear off, but she's never in a rush to get me out of the door.
Perks of small town living?
I think this is an issue people often overlook / does not get enough weight in the discussion.
I do wonder that, when such profound technologies (such as AI, social media) are rolled out, should they be subjected to studies from the "human perspective" for a longer period of time.
However, this might be impossible in the current system.
But this seems a bit click baity.
You can challenge your medical record, and presumably an AI transcription service would be in there. Unless .au is special in that way.
I can understand the appeal in the tech industry, where the increased costs can be deferred until the financial situation changes. But healthcare does not work like that.
I'd be hesitant to connect that to internal bleeding right off the bat though. Definitely would not trust claims that internal bleeding is a first-order effect of the drug.
My cousin used one of these apps with patient consent and said she then has to listen to the audio and rewrite it all. Stopped for that reason. Wasn’t even a time saver.
Strangely, I think Robin really just mistimed this. They stopped just as the state of the art came out and with their human in the loop transcription they might have been quite useful.
In fact, there will be MORE of those stories because it is shocking and strange, and clickworthy! The readers demand stories! But the reality is that there is no world where a story like this meaningfully informs the public about the accuracy of the services in question and the tradeoffs involved.
This is known as the long tail problem in ML, and it's a reality in almost every field. It's also why we don't officially have self driving cars, despite there being thousands of videos out there with cars driving autonomously for hours without any errors. But every now and then, there will be cases where the system doesn't work.
There are two interesting aspects here. One, we don't actually have quantitative data on how often this happens in the same field when human errors don't get caught. In a perfect world we'd have that, plus a long study for the "AI" systems, and we'd get to compare the two. Secondly, even if we'd have that data, people would still act out against "the machine" in the (ideally fewer) cases where it errors out, compared to a doctor. It's part human nature, part (manufactured) rage against the machines.
I also agree with your second paragraph. Case in point, when a waymo hit a cat, we got a shit ton of articles, riled up communities, and so on. Or every time that other car hits something we get plenty of press, even if some of the "accidents" are fender benders that likely wouldn't get reported otherwise.
So even if a system has the same rates as humans, if we can't solve for the mismatch in our ability to predict and recover from errors, final outcomes will be worse.
My experience dealing with ai-mediated processes is that the error recovery paths often simply don't exist, presumably because eliminating the personel that dealt with oddball and errors was the supposed benefit of having the AI deal with it in the first place.
If I report a bug in (for example) Slack that loses messages and cost me a lot of time and headaches, is it an appropriate response to say "but Teams has even more bugs" or "sure, but if you had that conversation face to face you might miss something too"?
And hence comparing AI use to a fictional situation where no errors happens is not meaningful.
I've personally caught multiple errors that were not just transcription errors, but elementary reasoning errors done by specialists I've seen - the baseline error rate from healthcare providers is far above zero.
I treat it as it is: a trendy hit piece against AI. There's nothing like "Why they still use Whisper Large V2?" in sight. If this article pushes the establishment to be more transparent about the AI tools, good. But right now there's not much to discuss.
Instead of relying on the one-shot transcription, have the system double check key facts by asking the patient for confirmation:
"Can you confirm you've you taken psychoactive drugs before?" – "yes/no"
Recently my partner received a prescription with instructions that were over the LD50 (we caught it as it was obviously too high). We reported it but it was likely the case of the clerk simply hitting the wrong button and not double checking the resulting sticker
I’d be interested to see the AI transcription failure rate compared with existing medical/pharmacy rates
It was listed on this wikipedia page until July 11, 2020. https://en.wikipedia.org/wiki/Preventable_causes_of_death
I know because I frequently cited it in arguments with citizen disarmament advocates before it was edited out.
I bet that to make the business case viable (or rather profitable), it's probably something small and cheap.
Bigger and better models don't come with any guarantees as to correctness either, but they do push down the probability of something as wrong as this happening by orders of magnitude.
That and processes. Even a smol dumb model can throw a report at both parties in the end where both need to sign off on it. Which should also scale better if both do, because the patient doesn't get fatigued because it's not happening many times per day.
Or you maybe mirror what humans do and ask for active confirmation the less plausible something sounds. Hmm. __
Point being that I wouldn't necessarily blame it on the tech itself, but rather the (probably) startup, the culture and the fact that no one is going to jail here.
Your honor, the floats are innocent. They were simply forced to do this by the evil startup founder.
Or at least my LLM thinks that you did.
Can we get there in the future - perhaps. But the accuracy needs to be much higher. I remember in the 90s when my father's clinic tried both Softvoice and Dragon. Comparing it to his receptionist typing out his notes, the accuracy was in the 80s - hence rejected. Just last week an AI startup here in Israel was kicked out of their HMO partner for having a transcription accuracy of only 92% in a mixed mode these where a doctor, patient, and caregiver are all in the room (think doctor + mom + kid)
For the curious - trained medical receptions have and maintain accuracy rates when measures of 97%. This is just transcription.
Hallucinating entire events is another, far worse thing.
The relevant information is the rate for each.
There are estimated hundreds of millions of medical transcription errors per year without AI. 42.4% of finalized medical notes still contained at least one error.
Prior to AI it would be about voice to text. Prior to that it would be some transcriber in India. Prior to that it would be the doctor themselves
but with AI you get to add "hallucinations" to the mix.
AI will too often not say "I dont know" and just make shit up instead.
However, the ai sounds positively heartbroken that it can't help, and the response latency is really good, so I guess that somehow makes up for the actual service being broken, right?
I feel like if you asked any half-intelligent AI "please review these notes and flag points we should review for correctness, or check with the patient?"
I'm pretty sure it would pick up a huge chunk of issues?
Have you ever actually read your notes on file?
This is basically living in a fantasy world. Errors are routine.
One of my notes in my current medical file states I injured my shoulder playing for a NFL team. I have never played football, and certainly not at a professional level. Fixing it is sort of like trying to fix your credit report - you supposedly get it done, and then 6mo later the same error pops back up again.
Ironically the reason this note exists is very similar to a way an AI scribe would misinterpret a conversation.
> Doctor: have you used any recreational drugs in the last six months > Patient: No > Doctor: < long pause as they review notes > > Patient (hallucinated most likely response): umm, actually there was one thing I hesitated to mention. Me and my girlfriends tried microdosing…
Well, doesn't every tool? The devil is in the details of just how easy it is to use correctly versus incorrectly, and what are the consequences if it's not used correctly. A tool that, when used as its manufacturer advises, fails as often as most AI systems do has no place in systems where peoples' health and safety can be harmed.
I’ve just had to call a support number for the first time in forever, and I’m met with a robot who perfectly understands my request (I speak very slowly and clearly), and proceeds to completely reject a 6 digit code, insisting it is 4 digits, and taking awkward silence to a new extreme (30-60 seconds between each interaction).
I’ve worked on speech-to-text and have read transcripts between two people talking where everything the one person says comes out like complete gibberish because they’re on the wrong side of a single-directional microphone. A human would interject either during the meeting, or at least while transcribing.
Putting it in production now is gambling your authenticity for cost saving, hoping your shitty voice bot is only as bad as the rest.
Hopefully with this oncoming mass unemployment caused by AI automation we will have some hands free to provide some genuine service instead of this ridiculous nonsense.
This one is kidney failure not bleeding around the kidneys but it still surprised me. Microdosing definitely makes it even less likely of course.
Sure, it sucks if your self driving car gets in a crash or your AI scribe incorrectly transcribes something to your medical record. This is news now. What isn't news is humans getting into crashes or doctors making poor medical decisions as a result of low quality or missing notes.
When doctors make mistakes, they can be held accountable -- their malpractice insurance rates go up, their licenses are subject to suspension or revocation, they can go to jail (eg if they are pill mills) or they/the practice get a bad review.
When AI makes mistakes, what happens? How is it held accountable?
This happens every single day and is effectively never reported. For far more nefarious reasons than a simple scribing error.
Drug seeking behavior enters notes all the time without much evidence and based entirely on a random doctor's (or even a triage nurse) hunch. A significant portion of those notes are outright false and incorrect. Once that is on your file and in a given medical system, you are marked for life.