Health NZ staff told to stop using ChatGPT to write clinical notes
rnz.co.nz
rnz.co.nz
Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off.
The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia.
I spoke to a practice nurse a few days ago to discuss this.
She said she didn’t think patients would care if they knew the data would be shipped off site. She said people’s problems are not that confidential and their heath data is probably online anyway so who cares.
If you're why doesn't this guy just check the AI scribe notes? Well, probably because with the amount of detail it writes, he'd be better off writing a quick soap note.
It's funny how the assumption is always that LLMs are very useful in an industry other than your own.
For all the whinging about bugs and errors around here the software industry in general (some niche sub-fields excepted) long ago decided 80% is good enough to ship and we will figure the rest out later. This entire site is based on startup culture which largely prided itself on MVP moonshots.
Plus plenty of places are perfectly fine with tech dept and the AI fire hose is effectively tech debt on steroids but while it creates it at scale it can also help in understanding it.
It is is own panacea in a way.
I think it is gonna be a while before the industry figures out how to handle this better so might as well just ride the wave and not worry too much about it in software.
Still software is not medicine even if software is required in basically every industry now. It should more conservative and wait till things settle down before jumping in.
Sure, we can eat that 80% if we made the debt and know where the bodies are buried. When AI does it, it’s more like “50% is good enough and hopefully someone smart enough can fix it up to 60% when it breaks”, which inevitably means we get it to 55%.
> while it creates it at scale it can also help in understanding it.
This feels like cope, and I’m not trying to be snarky. I also know that I had to train my brain to skip google’s “AI Summary” on searches because I’ve had a handful of wrong answers from it - not technically correct, not correct with caveats, just flat out wrong. So sure, AI could make a bigger mess and then we could trust it to help sort it out, but even if it finds three real problems and one non-existent one, it has still made even more work to sort out.
> Still software is not medicine even if software is required in basically every industry now. It should more conservative and wait till things settle down before jumping in.
Agree. We had a big company meeting about becoming an AI-focused company (we do healthcare-related software) and I’m honestly a little worried - I don’t use any AI in my work, and when I asked a colleague who implemented our new build process for help with migrating my own repo they said “I don’t know, I asked GPT to do it”. And that’s why the pipeline has a mega-long ternary for the name which doesn’t resolve, so all of our runs are titled “if [[ $matrixType === …” but they’re using AI, and they’re going to be celebrated for it
But engineers have these same problems. The key is that the content creator (engineers for codegen, doctors for medicine) is still responsible for the output of the AI, as if they wrote it themselves. If they make a mistake with an AI (eg, include false data - hallucinations), they should be held accountable in the same way they would if they made a mistake without it.
Are you willing to put your life in the hands of these people fully using the machines to do everything?
Acting like that smart people aren't getting one shot'ed by these machines is very dangerous. Even worse is how quickly your skills actual degrade. If knew my doctor was using anything LLM related, I would switch doctors.
I was updating some gitlab pipelines and some simple testing scripts and it created 3 separate 300+ line README type metadata files (I think even the QUCIKSTART.md was 300 lines).
That's funny. I would have said the same thing about your field prior to reading your comment.
[0] https://developers.openai.com/api/docs/guides/your-data#whic...
[1] https://developers.openai.com/api/docs/guides/your-data#stor...
[2] https://platform.claude.com/docs/en/build-with-claude/zero-d...
The models are licensed to Microsoft, and you pay them for the inference.
I've had similar experiences in Australia. I emailed one of my docs' practices asking if they use Heidi AI (or anything similar) and that I do not consent. They were using it without my consent.
In the consultation, he tried to give me the schpiel, including the 'it stays local' thing. The Heidi AI website has the scripts for clinicians; he ran through them all.
Oh, their documents for clinicians also mention every two sentences that patient/client consent is not required at all. I wonder why they keep saying that? Hmm.
This doctor knows I am a developer. When I asked him to explain what he meant by 'local data', he said the servers were in Australia. I almost flipped the desk. Aside from the fact that it is mandatory (it's the law! they do not have a choice!), it's ...kind of meaningless where the servers are, especially when he (on behalf of Heidi AI) was trying to sell it as a security or privacy feature. When I pointed that out, he just couldn't wrap his head around it. Of course he can't, he doesn't understand.
AHPRA's "Meeting your professional obligations when using Artificial Intelligence in healthcare" guideline[0] (not any kind of enforceable requirement, unfortunately) has great stuff in it. It encourages using it with the informed consent of patients. Even if my doctor read it and agreed with it, and cared about getting consent, how the hell can he inform patients sufficiently when he has absolutely no idea about, well, anything?
He keeps pushing it and asking me about whether I've changed my mind about allowing him to use it. No! He keeps asking me questions that only confirm he hasn't even done a perfunctory web search about why some people hate LLMs, especially in the context of PII and PHI.
I really do feel for clinicians, but these products are not the answer.
[0] https://www.ahpra.gov.au/Resources/Artificial-Intelligence-i...
I’d go as far as saying she’s right. And we’re in a tiny minority for even thinking about it.
That memo is how you make staff hide things instead of asking for help.
The scarier part though is that LLM-written clinical notes probably look fine. That's the whole problem. I built a system where one AI was scoring another AI's work, and it kept giving high marks because the output read well. I had to make the scorer blind to the original coaching text before it started catching real issues. Now imagine that "reads well, isn't right" failure mode in clinical documentation.
Nobody's re-reading the phrasing until a patient outcome goes wrong.
I wouldn't go as far as "no place in medicine" though. The Heidi scribe tool mentioned in the article is a good example, because in the end it's the doctor who reviews and signs off.
IMO the problem is AI doing the work with no human verification step, but I can 100% agree I don't want to have vibe-doctor for my next surgery/consult :D
I think any person using 'AI' knows it makes mistakes. In a medical note, there are often errors at present. A consumer of a medical note has to decide what makes sense and what to ignore, and AI isn't meaningfully changing that. If something matters, it's asked again in follow up.
You think wrong. I’m now encountering people on a regular basis arguing “those days are behind us” and it’s “old news.”
I think there is a chance that these systems will lead to a change where the note isn't the fundamental record of the encounter. Instead different artifacts are created specifically for each entity that needs it. Billing gets their view, and scheduling gets theirs, and, etc etc... It will, hopefully, give the practitioners a chance to get back to focusing on the patient and not ensuring their note quality captured one more billable code. Of course the negative is also likely to happen here too. As practitioners spend less time on the note they will likely not get that back in time with individual patients, but instead on seeing more patients. It will also likely lead to higher bills as the health systems do start squeezing more out of every encounter. There is no perfect here when profit is the driving motivator but with this much change happening I can only hope that it causes the industry as a whole to shake up enough to maybe find a new better optimum to land in.
This is what an EHR does somewhat. The discrete data elements in the DB and the way they are displayed in the system are a better record than free text notes.
The problem is creating standards so this data is easily exchanged. Anyone can read and parse a free text note - but if we had standards this would be less necessary.
We had to correct them at the end of the consultation.
This is just about not using free/public AI tools.
Heidi is frustratingly consistent at hallucinating stuff. I've seen it in almost all of the dozen or so summaries I've had from medical people recently (surgeon, physio, consultant). A GP I know tried for a month and then was like 'it's not worth the risk exposure to me or my patients'.
In the email I wrote out everything myself, absolutely no use of AI, but after I hit send I realised there was a pretty silly typo, nothing grave but it irked me.
I decided out of boredom to see would my email be considered AI as it was probably going to go through a million filters these days, I popped it into an online checker (I don't know the quality of these so who knows) and it told me with 75% certainty it was written by AI.
It was not at all. It was written overly hastily on a phone on public transport. So I wonder how someone who might be grammar orientated and particular with the semantics would prove otherwise.
I can see a company needing to find any excuse to let people go saying "well theAI says you used the AI to do your work, we're letting you go"
In fact, it's human transcribers who chose whether to forget the details of a case or whether to share the details of an especially funny patient with their buddies at the bar.
You can program a harness to always send a MEMORY.md file like OpenClaw, or use Vector Stores like OpenAI does, or find some other implementation of 'memory', but these are not an inherent feature of 'AI'. Quite the opposite...the LLMs we currently see will never learn or adapt by themselves, they don't touch their own weights
Enterprises are ok sharing their code base with OpenAI. I think it should be okay for patients.