I just hope someone is double checking whatever notes are taken. My experience with LLM tells me they're fantastically bad at remembering and even worse at producing factual stuff that can be used without context.
I suspect this will lead to a decline in accountability with there being another party to blame rather than the medical professional.
The LLM did it, not me.
Yeah no, I can tell you from experience with a clinic that things are checked. Let's talk about the real issues and how to enforce double-checking for people who would ignore it. Hyperboly like that is not helpful.
But, I wonder if those systems should randomly insert obvious markers to help here. "This has not been read." "Doctor was asleep at the wheel." "Derp derp derp." - like the fake gun images that the airport security has to mark.
I’ve literally been in a meeting with a doctor in the last week who wrote something down wrong on the damn computer in front of me. And I’m talking a specialist consultant. I’m sure if I didn’t mention it that the incorrect data would be checked over and over again to make sure it was correctly incorrect…
But then, there are secondary effects like how much time will your doctor have and how ready they are. In practice the notes take time. If you're unlucky, you're going to be late, on a busy day, and your notes will be done many hours later from recollection. In that case, even an imperfect system can increase the overall quality if it enables faster turnaround. I know of cases where the automatic notes generation did catch issues which the doctor simply forgot about.
The individual stories are brutal, but overall they say very little - was that the only mistake that doctor made in their life, or are they making 10 a day? In general we have to accept mistakes happen and build a system that catches them or minimises the impact.
Far be it from me to suggest that doctors aren't both fallible, and subject to arrogance that makes it harder for them to catch their mistakes—what highly skilled professionals are immune?—but "doctors make mistakes" is, while doubtless completely true, a very different claim from "doctors don't check things."
I’m hoping for a lot of legal precedent showing that an AI cannot be blamed, especially in a medical context.
How would that even work?
So far we're doing pretty good with that idea globally (I've not seen any case going the other way in court)
Cory Doctorow wrote about it a while back. I think it was this article "Humans are not perfectly vigilant" [0]. It explains how technology is supposed to help humans be better at their jobs, but instead we're heading in a direction where AIs are doing the work but humans have to stand beside them to double check them.