AI assists clinicians in responding to patient messages at Stanford Medicine
med.stanford.edu
med.stanford.edu
Here’s how I see this playing out: practices start to use LLMs to answer patient queries to save money. People realize their queries aren’t actually being answered fully (or at all?) and stop contacting their doctors at all. Practices save money and patients get worse outcomes. Exactly as we’ve seen in customer service over the years - talking to a human is nearly impossible.
An alternate possibility is the LLM gives such bad advice that it causes irreparable harm to a patient and nobody can be held liable because the doctor didn’t issue the advice and neither did the company which produced the LLM.
The universe where an LLM used in this situation does anything positive for the patient is vanishingly small. But I’m sure providers and practices will love it, since it effectively will allow them to not do part of their job and increase margins respectively.
I think someone will be liable; this has already been tested in at least one court with something less life-critical: https://news.ycombinator.com/item?id=39378235
Doctors will have more time for real questions if they aren't answering idiotic questions. You are living in a bubble.
https://www.nbcnews.com/health/health-news/doctors-move-end-...
> When Melissa Boughton complained to her OB-GYN about dull pelvic pain, the doctor responded by asking about her diet and exercise habits.
> On this occasion, three years ago, the OB-GYN told Boughton that losing weight would likely resolve the pelvic pain. The physician brought up diet and exercise at least twice more during the appointment. The doctor said she’d order an ultrasound to put Boughton’s mind at ease. The ultrasound revealed the source of her pain: a 7-centimeter tumor filled with fluid on Boughton’s left ovary.
It’s interesting that you frame it as “faulting” the patient - if a patient comes in with shortness of breath, should doctors avoid asking patients if they smoke? If they ask a patient about previous surgery, are they blaming the surgeon? If a doctor asks about family or work history, are they criticizing family or work choice respectively?
https://www.nature.com/articles/s41591-020-0803-x
> Evidence suggests that physicians spend less time in appointments and provide less education about health to patients with obesity compared with thinner patients, and patients who report having experienced weight bias in the healthcare setting have poor treatment outcomes and might be more likely to avoid future care. Obesity also adversely impacts age-appropriate cancer screening, which can lead to delays in breast, gynecological, and colorectal cancer detection.
> Negative influences on engagement with primary care were evaluated and ten themes were identified: contemptuous, patronizing, and disrespectful treatment, lack of training, ambivalence, attribution of all health issues to excess weight, assumptions about weight gain, barriers to health care utilization, expectation of differential health care treatment, low trust and poor communication, avoidance or delay of health services, and seeking medical advice from multiple HCPs.
Ozempic actually is effective, and will likely lead to significant improvements.
There is no scientific evidence, whatsoever, that diet and exercise is an effective way of losing a clinically significant (>5%) amount of weight and keeping it off for a long period if time (5y). Go ahead and try and find even one study that shows this is the case.
When you diet and exercise, your basal metabolic rate slows down as much as 20-30%, permanently, and your hunger increases. Your BMR is where the vast majority of your energy expenditure goes, no matter how much you work out. In fact there's reason to think that more exercise will actually slow your BMR. Body weight set point is principally genetic and epigenetic, as evidenced from twins studies.
Maybe we'd make some progress on this particular topic if we stopped throwing out tired tropes and blaming people. The only scientifically proven methods of achieving significant, long-term sustained weight loss for most people are GLP-1/GIPs or bariatric surgery (but even there, only a gastric sleeve or roux-en-y work, lap bands do not).
Here's a 29-study meta-analysis which walks you through what I said in more detail [1] and of course the famous Biggest Loser study where everyone on that show regained all the weight in the six years following. The more they lost on the show the more they gained back. [2]
Let's not even get started on the use of insulin to treat type 2.
It's pretty wild how backwards our approach to metabolic health is from a clinical perspective. Your response here is a perfect example.
Now look at Tirzepatide. 90% success vs. 5% success. [3]
[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5764193/
[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4989512/
[3] https://www.nejm.org/doi/suppl/10.1056/NEJMoa2206038/suppl_f...
Or the fact that the entire sales pitch is “this will be cheaper than actual people who know what they’re doing”, maybe a side claim of “outperforms people on our training set” coupled with “we accept no liability for inaccurate advice or poor outcomes”.
The sole purpose of programs like these is to save money. It has nothing to do with helping anyone. The goal is to sound like your system is providing the service people paid for, without actually spending the money to provide that service.
I will believe a company will is doing the right thing when the developers personally accept the criminal and legal liability that an actual doctor or nurse providing the service is subject to.
But then to add insult to injury this just further increases the wait time for people by forcing them to deal with the “AI” before they get through to the further reduced pool of actual specialists and correspondingly increased wait times there as well.
This is not rocket science to understand, and literally nothing I’ve said here is new - not the clear prioritization of cost cutting, the overselling of “AI”, and the generally unsafe use of randomized statistical models to provide factual information.
My question is why you think that this time is different vs literally every other example?
I'm as skeptical of GenAI as they come, but let's not pretend that the people working on GenAI customer support are the same people who should be working with the AMA to address the artificial constraints in medical education.
The thing I notice about every argument where I see "we can do both", is that in reality we never actually do both
That's why people get so adamant about choosing their way instead of the other way. There's always two options, we could do both, but we never, ever seem to
Let’s summarize the situation here. A group of people have chosen to work on GenAI customer support. You’re criticizing them for not working on a difficult political problem instead.
Meanwhile, what are you doing? Which of the political activist groups working on the problem you care about so much have you joined? How have you offered your support?
If this issue is so important to you that you think “we” should be devoting our full attention to it, surely you’re out front leading the charge, right?
>You don't think there's any conceivable way that this may actually help people?
I don't. I believe that exactly as OP described people will notice they are talking to some AI and just give up, as they correctly understand that the AI does not have the abilities and understanding that might actually help them.
Health things around here have legal restrictions that definitely prevent them from using GPT, thankfully.
This is why people are pessimistic about ML/AI in people-facing positions. It has happened with automation in the past, it is _currently happening_ with ML/AI systems in customer-facing applications, and the momentum of the field is headed in that direction. To me, it would be very silly to assume that the field would make a sudden 180 from decades of momentum in a rather established direction, with no established reason other than the potential for it to be good.
This is why people tend to generally be upset about it, is my understanding. It's a good tool, and it is oftentimes not used well.
I don’t interpret the OP as slamming AI. It’s criticism of the way in which companies find ways to save a few dollars.
https://www.npr.org/sections/health-shots/2013/02/11/1714096...
> He took a picture of a man in a gorilla suit shaking his fist, and he superimposed that image on a series of slides that radiologists typically look at when they're searching for cancer. He then asked a bunch of radiologists to review the slides of lungs for cancerous nodules. He wanted to see if they would notice a gorilla the size of a matchbook glaring angrily at them from inside the slide.
> But they didn't: 83 percent of the radiologists missed it, Drew says.
At the end of the day the metric of success for these doctors is to find cancer, not impossible images of gorillas.
The study is kinda “funny”, but it doesn’t really mean anything. Even the article itself does not claim that this is some kind of failure
https://cognitiveresearchjournal.springeropen.com/articles/1...
> For over 50 years, the satisfaction of search effect has been studied within the field of radiology. Defined as a decrease in detection rates for a subsequent target when an initial target is found within the image, these multiple target errors are known to underlie errors of omission (e.g., a radiologist is more likely to miss an abnormality if another abnormality is identified). More recently, they have also been found to underlie lab-based search errors in cognitive science experiments (e.g., an observer is more likely to miss a target ‘T’ if a different target ‘T’ was detected). This phenomenon was renamed the subsequent search miss (SSM) effect in cognitive science.
The point is, an image of a gorilla is literally completely impossible and has no relevance. That’s why a security vuln is not analogous
There's a risk to "rule out pneumonia" style guidance resulting in a report that misses these things in favor of "yep, it's pneumonia".
As someone who is trained in and comfortable reading radiographs but is not a radiologist, I can tell you that putting a gorilla on one of the views is a poor measure of how many things are missed by radiologists.
Effectively interpreting imaging studies requires expert knowledge of the anatomy being imaged and the variety of ways pathology is reflected in a visibly detectable manner. What they are doing is rapidly cycling through what is effectively a long checklist of areas to note: evaluate the appearance of hilar and mediastinal lymph nodes, note bronchiolar appearance, is there evidence of interstitial or alveolar patterns (considered within the context of what would be expected for a variety of etiologies such as bronchopneumonia, neoplasia, CHF,...), do you observe appropriate dimensions of the cardiac silhouette, do you see other evidence of consolidation within the lungs, within the visible vertebrae do you observe appropriate alignment, do the endplates appear abnormal, do you observe any vertebral lucencies, on and on and on.
Atypical changes are typically clustered in expected ways. Often deviations from what is expected will trigger a bit more consideration, but those expectations are subverted during the course of going through your "checklist". No radiologist has look for a gorilla in their evaluation.
It is pretty clear that the layperson's understanding of a radiologist being "look at the picture and say anything that is different" is a complete miss on what is actually happening during their evaluation.
It's like if I asked you to show me your skills driving a car around an obstacle course, and then afterwards I said you are a bad driver because you forgot to check that I swapped out one of the lug nuts on each wheel with a Vienna sausage.
Things like obvious fractures left out of a report.
https://cognitiveresearchjournal.springeropen.com/articles/1...
> For over 50 years, the satisfaction of search effect has been studied within the field of radiology. Defined as a decrease in detection rates for a subsequent target when an initial target is found within the image, these multiple target errors are known to underlie errors of omission (e.g., a radiologist is more likely to miss an abnormality if another abnormality is identified). More recently, they have also been found to underlie lab-based search errors in cognitive science experiments (e.g., an observer is more likely to miss a target ‘T’ if a different target ‘T’ was detected). This phenomenon was renamed the subsequent search miss (SSM) effect in cognitive science.
Putting a gorilla on a view and then posting to NPR a sensationalized article about how "83% of silly radiologists just can't see what you managed to see" is not that.
In fact I would argue the SSM effect is present in many aspects of medical decisions and likely other industries. The other way to frame the SSM effect is to call it the "this case has the initial patterns of the thousands of other routine cases of disease X, so it is almost guaranteed to be disease X and I have to get home for my daughter's dance recital." effect. It's an optimization strategy that works most of the time.
Interpreting X-rays in isolation has little relevance to actual clinical practice.
As helpful as it is for rads to catch unexpected incidental findings, that's not the point. If I'm waiting on a stat read, I've got specific questions I'm looking to have answered quickly. I literally don't want the radiologist wasting time hunting for random oddities if it delays getting the reads back.
So far, the code it produces looks really good at first glance, but once I really dig into what it's doing, I find numerous problems. I have to really think about that code to overcome the things it does wrong.
I think many doctors are used to doing that already, since they have to correct their interns/assistants/etc when they come up with things that sound good, but ultimately aren't. But it's definitely something that they'll fail at sometimes.
In that situation it’s extremely easy to just blindly trust an “expert” third party, particularly one where you can’t ask it questions about how and why.
> Barriers to adoption include draft message voice and/or tone, content relevance (8 positive, 1 neutral, and 9 negative), and accuracy (4 positive and 5 negative).
That's it! Outside of that it looks like almost all the metrics are about how this makes life easier for doctors. Maybe there is more in the supplement, I have not read closely.
Look: I am not a policymaker or a physician. It's quite plausible that a few LLM hallucinations is an acceptable price to mitigate the serious threat of physician overwork and sleep deprivation. And GPT-4 is pretty good these days, so maybe the accuracy issues were minor and had little impact on patient care.
What's not acceptable is motivated reasoning about productivity leading people to treat the accuracy and reliability of software as an afterthought.
It's also worth noting that, from the abstract, the study's objective isn't to study the LLM's accuracy. This is a study of the effectiveness of this drafting system's implementation in the hospital. I'm not saying the accuracy isn't an important component of the system's effectiveness, but it's not the question they're answering.
Say I released a linear algebra library and extolled its performance benefits + ease of development, then offhandedly mentioned that "it has some concerns with accuracy" without giving further details. You wouldn't say "ah, he focused on performance rather than accuracy." You'd say "is this a scam?" It is never acceptable for healthcare software researchers and practitioners to ignore accuracy. LLMs don't change that.
The only thing that sort of cynical laziness is good for is justifying bad decision-making by doctors and hospital administrators.
There's not a singular metric like accuracy that summarizes the quality of a generative system. What would an "accurate" message be? I can think of several aspects: mimicking the doctor's tone, addressing the correct patient concern, not providing clinically false information, not being too long, using the correct language. A good statistical characterization of this system requires rating multiple aspects of the content. These researchers measure the system's quality through user surveys and interviews. Is there a better ground truth to compare it to?
There's an important distinction between accuracy and effectiveness. These researchers are interested in the system's effectiveness. An algebra library can't be effective without being 99%+ accurate. Plenty of tools can still be useful without being perfectly accurate. Diagnostic tests are a great example. Conversely, you can have tests that are 99% accurate, but not useful, depending on the context and implementation.
I think that's why these researchers are less concerned with accuracy. Effectiveness is a separate concern from quality/accuracy and can be studied separately. They're different research questions.