Google AI has better bedside manner than doctors – and makes better diagnoses

Odds are you don't even see a GP anymore in the United States, you see a Nurse Practitioner who then potentially forwards your information to the doctor. Most visits your doctor spends less than 5 minutes on you.

This assumes you even have a GP already, since most are booked out 3 months or more for new patients. Reality is, the AI doctor you CAN visit is better than no visit at all.

PCPs are notorious for misdiagnoses, they're expensive, hasty, often times don't believe or listen to patients, and frequently just don't have all the data. This isn't necessarily their fault -- they're overworked and the medical industry isn't making things better -- but the reality is primariy care isn't working very well right now even in developed countries. Imagine in developing ones...

Diagnosing a patient is in many ways an expert system problem which computers are excellent at. Amassing the data from every medical textbook ever written, plus every study ever done, plus clinical conversations with patients and their medical history (the hardest part), and you have the best PCP ever made. Add a nurse to manage the physicality of it and one day connect data to the system to track lifestyle behaviors, sleep, and things people won't necessarily self-report, and you have something revolutionary. And AI is kinder and more compassionate, as the article said.

No wonder people have been trying to crack this nut for decades (albeit with minimal success). I hope the LLM revolution helps make another big round of progress.

> It hasn’t been tested on people with real health problems — only on actors trained to portray people with medical conditions.

Obviously the FDA won’t allow them to operate in real patients, but designing a system that knows how to identify which script you had your actor play is very different from designing a doctor.

> an LLM has the unfair advantage of being able to quickly compose long and beautifully structured answers, Karthikesalingam says, allowing it to be consistently considerate without getting tired.

The last thing I want is to read through pages of AI blogspam at the doctors office. Where’s the AI that takes its pages of fluff and boils it down to the point?

That said, I am cautiously optimistic, as there’s definitely room for improvement. I’m lucky to be basically the same demographic as doctors, so they tend to actually listen to me and believe me. I know from second hand experience that for folks who don’t look like their doctors, getting one to take them seriously can be herculean. One individual I know had the UCLA (male) medical staff call security to escort her out of the building because they thought she was being too hysterical and should just tough out whatever it is she was dealing with (they refused to look). She then went to a (female) private practice gyno who took one look and immediately saw there was indeed a massively pain-causing complication present.

Let’s hope the AI doesn’t train on the wrong transcripts though…

Aside, the best part is UCLA didn’t want to let her graduate until she paid the medical bills from that “visit”. She ignored every one and they seemed to forget about it. Medical incompetence has its upsides.

Aside aside, this is also why “Shouldn’t the best test scoring individuals be admitted to (medical) schools, regardless of demographics? To hell with diversity, you want the smartest doctors possible, don’t you??” is flatly invalid.

We've got ardent supporters and detractors, and often with this type of divide reality is somewhere in the middle. This is a huge, dangerous, and sometimes hard to understand technological development.

maybe if you are rich. For most people medicine is going to a GP once or twice a year, spending an hour in a queue, then seeing a doctor for 10 minutes.

In general looks like diagnosis will be in large part overtaken by ML, the amount of knowledge you can cram into it is orders of magnitude more than with the smartest humans. And doctors (especially good ones) are very expensive, with this even some mediocre one can produce amazing results.

“the base LLM with existing real-world data sets, such as electronic health records and transcribed medical conversations”

So it trained on text that a healthcare provider extracted from the HPI/interview, labs/tests, radiology etc?

So it’s just a diagnostic assistant that can chat? Is it going to get conversational info out of non-English speakers and indigents? The urgently ill who are unable to vocalize what they’re feeling? The delirious who have “altered mental status”?

What about the huge amount of geriatrics who have significant cognitive decline and lack the ability and vocabulary to deliver information? And this is going to bother the terminally online - but what about the low SES who speak in virtually indecipherable urban patois?

This piece of shoftware isn’t going to help anyone. It’s going to be for articulate young to middle age educated people who don’t want to talk to a chatbot. And they’ll probably hate using it as they chat with it for filling out visit screening and new patient forms. If it ever deploys.

An “underserved” patient may use it out of desperation to get what they need but I would never in a billion years want this for a patient I gave a shit about.

I guess there’s the crux. They don’t give a shit. Some PE group will get it and you can talk to it before your virtual visit with your human doctor on Zoom. It’ll be like Elysium. Maybe I’m bitter because I still commute all over town. Maybe that’s what all you software people want, you like WFH so much, let the doctors do WFH too eh? Talk to the chatbot, you guys made it and it’s better than us medical cartel types anyways.

> An artificial intelligence (AI) system trained to conduct medical interviews matched, or even surpassed, human doctors’ performance at conversing with simulated patients and listing possible diagnoses on the basis of the patients’ medical history1.

> The chatbot, which is based on a large language model (LLM) developed by Google, was more accurate than board-certified primary-care physicians in diagnosing respiratory and cardiovascular conditions, among others. Compared with human doctors, it managed to acquire a similar amount of information during medical interviews and ranked higher on empathy.

So, they created an interview machine? Getting better than a medical doctor at conducting a dialogue based interview is simplistic and sophmoric.

Primary Care docs are the most generalist doctors that exist, and medicine is very much a specialist field.

2nd, doctors do not rely on only dialogue to evaluate a PT, but instead they use in situ observations and sensing devices.

On the good side, this could probably help replace some Nurse Practicioners working for insurance companies that have no business getting anywhere near a PT.

That being said, I had to laugh, this is one hyperbolic headline. There are of course some caveats.

"Few efforts to harness LLMs for medicine have explored whether the systems can emulate a physician’s ability to take a person’s medical history and use it to arrive at a diagnosis. Medical students spend a lot of time training to do just that, says Rodman. “It’s one of the most important and difficult skills to inculcate in physicians.”

This does take a lot of training and experience. The challenge in diagnosis isn't about asking a history and integrating the information, it's about effectively encouraging patients to provide necessary information that they might not realize is relevant or know how to articulate. Different patients often require wildly different approaches. Medical literacy does play a role, but a patient that would say, "Hi doctor, I experienced central chest pain accompanied by discomfort in the upper stomach that happened two hours ago" (from pg 33 of the preprint) is not realistic. More likely you get a vague complaint of "heartburn that started a while ago".

Similarly: "Currently, I'm not on any prescribed medications". More frequently you get something like "I take a blue one" or "Golly Telly" (Go-Lytely) or "Gabatini" (Gabapentin). I do think an LLM could probably parse these but such idiosyncrasies in a history do compound. And though someone may be prescribed a med and think they take it as prescribed, sometimes it takes a hunch and clinical experience to tease out that in fact the med is not being taken at all as indicated.

Moreover, the better bedside manner was assessed via text conversation. I wouldn't quite call that "bedside" manner. I also wonder how such a system will deal with patients that have self-diagnosed themselves and looked up the "right answers" to get what they want -- a difficult reality that takes some parsing to figure out what's real and what's not.

Overall though, the preprint, in contrast to the Nature news article/headline, does a better job of discussing these limitations. I congratulate them on excellent work. Thank god LLMs can't do surgery, though I'm sure my time will come as well.

What? What kind of "bias" would this entail? Or is this more of the "ethics" garbage that is used to lobotomize LLMs? The real scientific-technical problem and discussion about aligment (e.g. paperclip maker) has been replaced by some strange grift that opens doors for people without any useful knowledge into the New Big Thing where the money is.