Nurses Say Hospital Adoption of Half-Cooked 'AI' Is Reckless | Hacker News Reader

Nurses Say Hospital Adoption of Half-Cooked 'AI' Is Reckless

techdirt.com

111 pointsrntn79 comments

Whole areas of publishing are trying “have an expert write a book mostly with AI, then have an editor un-fuck it”. There are tons of businesses trying this “brilliant” plan right now.

From what I can tell, the labor savings is… low, and maybe zero. The output’s such trash, it takes enormous effort to clean it up. This stuff only seems to be a huge time-saver when someone doesn’t give any shits about how good the output is—so, mostly spam, scams, and astroturfing.

[edit] to temper the pessimism, it is really nice in small ways, in capable hands. But it’s not gonna be replacing many workers any time soon. Outside helping editors and writers with idea generation and little touch-ups, it’s great for non-writers who want to maintain a consistent tone in e.g. marketing copy, but requires the operator to have good taste and to be fairly thoughtful, which means they’ve got to be above-average at this stuff to begin with.

It's more a statement on the time than the technology, but at this moment GenAI was destined to become a force multiplier for laziness, grift, profit, and predation more so than forces that will make society better.

The second order effect is that because this effort is now possible every shadow or unknown contains this effort. Weirdly worded Reddit title? Must be an LLM. Poorly shot image? Must be StableDiffusion.

> StableDiffusion

New underground marketing campaign by Polydactyls Against Mutilation, or lazy excuse when you get caught using the machine?

Well before GenAI, CStross has blogged about the value-add of editors being quite large.

I suspect he probably can't give a complete answer to the relative expense of human editors for human vs. AI writing given that he's said that the merest hint of GenAI in his profession would be a reputational death sentence.

But he posts here sometimes, so: *waves*

LLMS, like cryptocurrency has now been discovered by the business bros with the dollar signs for eyes. This will make a big part of the world will get actively shittier and aside from actual cool and useful applications like summarization etc. I predict that the malicious usecases outweigh the good ones at least 10:1.

FTA - "The problem is that healthcare giants’ primary obligation no longer appears to reside with patients, but with their financial results", that, in a nutshell, is the core of everything wrong with US healthcare: it focuses first on shareholders, and secondarily on patients. It's no surprise the US spends more on healthcare per capita than any other country in the world and yet gets far worse outcomes.

This is a case where LLMs and similar generative approaches are outright dangerous and damaging. Not so much immediately but in that they erode trust in machine assisted diagnosis or treatment and set back adoption of things that could actually work substantially.

Namely tools like expert systems which are uniquely suited for the application can easily be confused with the current batch of "AI" despite being much more sane and useful.

I have a background in expert systems in a medical context, and it is wild to see people reaching for LLMs in this space. Having a clear audit trail for how a decision was reached is a requirement, and these tools don’t provide it.

LLMs can be a component in a system. See them as better language handlers than existing algos for looking at what doctors write to see if there are mistakes or omissions. Flag things for review rather than make decisions. In the audit trail, there is less concern about what model you used to flag something for human review. If you can catch mistakes, you can save lives, and the hospital money

Can you provide more context on what you do? I’m curious about why you don’t think an LLM can provide that audit trail!

I'm a service dog

     error-prone automation, programmed from the ground up to prioritize money over health, is incorrectly denying essential insurance coverage to the elderly.

I remember being interviewed once for an opportunity in a startup working in developing a product that using AI they would automate answering to incorrect insurance denials (which is a tedious process).

AI or not, the problem of incorrect denials is so bad that there is a market for it.

> AI or not, the problem of incorrect denials is so bad that there is a market for it.

Who is the customer? Insurance companies?

Smaller clinics that would feel heavy to have one hire full time doing just that.

Yea there are cottage industries on top of cottage industries in the medical insurance space. First you have the first order: DR supplies service, bills patient’s insurance, then you have the hospitals who outsource all billing, those outsourced companies outsource the claims management, which outsource the claims response, and the exact same structure exists on the insurance company side. And I’ve simplified a lot… there are coder companies, insurance coder consultants, etc. it’s totally wild.

If you ever wonder why. US medicine is so expensive: it’s bec there are 100 hands all reaching for their piece of your $120 annual checkup.

Or put another way, HIPPA is nice and all, but by the time all is said and done, 8-10 pairs of strangers eyes have seen your vagina fungus infection pharmacy bill.

I once was lucky enough to have my primary care physician through a university teaching hospital that outsourced almost nothing. Billing was in-house, insurance ops and claims processing was in-house. Labs went to an accredited in-house reference laboratory.

I am sure it was easy to justify doing everything in-house because the university held accredited degree programs for virtually every healthcare area from medical school to ophthalmology and optometry, physical therapy, pathology, dentistry, a college of nursing, pharmacy school, as well as specialties like oncology, cardiology, and anesthesiology. For the areas that weren't medical disciplines, they operated degree programs for bioinformatics, hospitality (the hospital cafeteria was excellent), and several other areas that just so happened to address essentially all the needs of a modern healthcare organization.

It was a fantastic model, but it might have only made sense in the context of a large teaching hospital.

This will be just like a self driving car system which I will not name, where you are handed a giant footgun with the disclaimer that you are responsible for any mistakes or hallucinations it might make.

As a healthcare tech investor, I do see a lot of startups selling potentially dangerous AI systems into the healthcare system. That said, there are also a good number of companies that are implementing systems thoughtfully to address a number of issues that are very real in healthcare like staff burnout, continuing education, adherence to standard of care, managing complex value-based payment contracts and coordination of care, etc. The trouble I see is that clinician/hospital buyers of these systems can't always tell the difference. A basic initial filter that can be used is simply (a) does the team have an experienced medical professional with power on its executive team, and (b) does the team credibly know how to measure clinical quality impact of what they're building and do they have a plan to honestly measure it.

Where do you see it being added thoughtfully?

Oh it just so happens to be his portfolio.

Is it good because it's in his portfolio, or is it in his portfolio because it's good?

Fall prevention to start. This is relatively benign and a good place for health staff to gain comfort around the technology.

Many nurses already have an internal sense when a model might be “off” for a patient, so it’s easier to know when it’s predicting accurately or not.

(1) https://www.frontiersin.org/articles/10.3389/fmed.2023.12851...

Actual articles from April:

Nurses gather at Kaiser SF to protest AI in health care

https://www.nbcbayarea.com/news/health/nurses-kaiser-sf-prot...

A.I.’s impact on nursing and health care

https://www.nationalnursesunited.org/artificial-intelligence

I was working at a big tech company and was critical of using AI in healthcare because of the hallucination problem. I was quickly fired for being 'too negative' and 'not showing a growth mindset'.

And this is why we're doomed as a species.

How many posts on yesterday’s “Who’s Hiring?” were for exactly this tech? Think I counted at least 5.

    "No computer, no AI can replace a human touch,” said Amy Grewal, a 
    registered nurse. “It cannot hold your loved one’s hand. You cannot 
    teach a computer how to have empathy.”

Sadly, this is terrible tactics. The people they are protesting do not see lack of empathy as a problem and probably even see it as a feature, as empathy is a bug from their point of view.

They'd be much better off talking about how obviously bad the care being given is and how easily people will be able to sue them for criminal negligence. And also, I've heard, you know, not that I'd ever do this, but I've heard some other nurses are even starting to tip off the patients that this is something they should sue over and giving them pointers on what to ask for during discovery. Certainly not something I'd ever do and I don't know any nurses personally who do this. I've just heard rumors. If you get my drift.

I'm not celebrating it, just calling it like it is.

Does anyone care what goes on in the mind of a person making a healthcare LLM product? I think we all know what they’re thinking: ride the hype train; get paid.

I think the nurse here is trying to appeal to human beings who are, more or less, afraid of dying alone.

One of Peter Lee's arguments in his AI in medicine book[1] is that the GenAIs (GPT4) actually excel at empathy. He gives a pretty compelling example where the GPT is able to empathize very well with a young girl who is having a medical issue. Empathy is part of the training set.

[1] https://www.amazon.com/AI-Revolution-Medicine-GPT-4-Beyond

This strikes me as something that will fade over time, though. We will eventually learn to recognize fake empathy, just as once upon a time when a corporation said "Your business is important to us and we're trying to get a support person on the line for you as quickly as possible", it was believable and there was a good chance your customer believed it. Now of course we've all got a pretty good idea it's not true.

An AI can not empathize. We don't even really want it to; who wants to build an AI that "really" experiences losing a limb or losing a daughter? Not anyone I want actually building AIs. So this isn't even about whether they're "really conscious" or any of those somewhat tedious debates; even if they are human-level AI already they literally can't empathize. See the recent article where Meta's overly helpful AI yielded an answer as to how New York's public schools treated its disabled child. Even if the text was completely accurate it still had no standing to emit such text.

Kaiser has been working with Stanford for years, to train different models, especially with images. A good use-case could be for diabetic retinopathy. There are more and more diabetics and this can affect vision over time, if not properly diagnosed.

Kaiser has ramped up and now takes more and more eye pictures to check for diabetic retinopathy. But it is a slow and manual process, requiring optometrists and ophthalmologists to seat and careful check pictures one by one (takes about 20min per picture). It is just possible to screen as many as we should screen. Here AI could really help, and pre-scan pictures for special attention, and be able to screen order of magnitude more patients.

I guess at the end it might be something like should we just screen 10 patients with a doctor, or pre-screen 10,000 in that same time, and have the doctor still see 10 patients ?

we are assuming that the state of health care technology somehow starts with high data integrity that isn't reckless to begin with anyways. As a technologist, it pains me to say that medical personnel having a healthy sense of interpretation and distrust for the system is a good thing.

So where's the urgent legislative push to stop this?

That's not possible, there's too much money to be made from the credulous.

If AI is telling the nurse the substance and quantity I should be injected with I am going to have some concerns. If it is filling forms and unburdening doctors and nurses from bureaucratic busy work then it is great.

Startups are treating LLMs like a silver-bullet that can be applied to every area. They are a specific tool that do very well at specific problems (for now).

Ultimately the substance and quantity you will be injected with also depends on those same bureaucratic busy work. If the LLM hallucinates a new condition for you when filling in some field, or if it gets your BP slightly wrong, you're in for a bad time.

Are we going to place any blame on the people selling this “AI” and their ridiculous claims? Sam Altman in particular?

Where is the evidence that a half-cooked reckless AI or an LLM for that matter was rolled out in this hospital? It’s not like this is the first AI tech used in a medical environment. Patients deserve the best care possible regardless. Who would argue that the current system is working as best as it could?

Anything which has clear paths to decision making, does not need AI.

Anything which requires only massive data processing, does not need AI.

Regarding the adoption: The crux is mentioned in the article well. You are building on top of faulty systems. However great the AI model is, you are given faulty data to process. It will not end well.

I would say the only reasonable and logical explanation is that these nurses are just salty that they have missed the AI stock craze and they just want to generate a dip for a buy-in. Gotta look out for those 4d chess moves. taps head (/joke <- just in case)

Something AI people have known forever is to not use an AI solution where the cost of a misprediction is large.

Tech Bro: "Look, I told it I had a headache and it told me to take some Ibuprofen. Pretty soon you can fire all your doctors and nurses since DocGPT will do it all!"

And eventually: "Our other product, InsureGPT, has been specifically trained to identify legitimate expensive claims that can be denied to maximize the probability of exhausting the patient in the appeals process!"

This is so familiar it hurts. I worked on something which originally had the goal of giving doctors more time with patients, and it was slowly but surely steered towards being a useful tool for insurance companies. I was too naive to see it coming.

InsureGPT would actually be great at its job. In the same way LLMs can never say "I don't know" it could never admit the claim was valid.

Former nurse turned dev.

I see a few of these AI note taking apps for healthcare and can’t help but shake my head.

This is just not how nurses work on the ward. Nurses are most worried about liability. So notes are written in a way to protect ourselves in addition to actual patient info. Probably more so now that I think about it.

I'm sure workers don't want their jobs automated. Its bad for business.

Tbh, I had enough incorrect doctors in my life, I'd ALWAYS want to see what the AI says.

Why would AI provide better diagnoses if the training data comes from humans?

Humans have bad memories and get tired.

Reply on news.ycombinator.com