Comment by pintxo | Hacker News Reader

Comment by pintxo | Hacker News Reader

Parent Full thread

> ... it means that you are probably better off with GPT than with a real doctor if you don't have something super peculiar that would take even true experts a lot of close attention and research to figure out.

I would argue the opposite.

Any reasonably well trained AI should outperform doctors on rare/exotic things, that most people will see never or only every other year. As it will still be described in the literature, the AI will have picked it up, but our average doctor is more likely than not to have forgotten about it since med-school, if she ever heard about it in the first place.

But when it comes to the normal day-2-day stuff, I am more afraid of the AI having a sudden case of hallucinations vs the doctor who probably has seen that same case yesterday and might not apply the latest state-of-the-art treatment [1], but it will probably be fine.

[1] It often takes 5 years or more for the medical state-of-the-art to transpire into daily practice "in the field".

Edit: Add citation and fix typo.

>vs the doctor who probably has seen that same case yesterday

This is actually super dangerous and probably the cause for a lot of misdiagnosing and medical errors these days. From the paper, it appears that GPT does much better in these cases.

This study does not test the doc's 'feel'. That's how an experienced doc evaluates patients, in good part. You develop an unconscious 'feel' for the stuff you see often. So, calling this study a fair comparison is actually pretty laughable.

This "feel" is just experience - aka having seen many things without remembering all the circumstances. It is nothing supernatural.

Well, yes that's my point. You deprive the docs of most stimuli they really rely on, and call it a comparative study. It's like saying A* is better at finding its way into a maze than a blindfolded man, and call it success.

Which is how lots of research, including medical/pharmaceutical [1] studies, are done.

[1] Sure, you have to test your new cancer drug on actual cancer patients, but why include the 65 year old obese patient, when you could instead include the 32 year old triathlon athlete into you cohort?

>like saying A* is better at finding its way into a maze than a blindfolded man

It's not just better than a blind man. It's better than any human, once you scale the task up beyond a certain point. Consequently, we should not get hung up on the path taken if the ultimate goal is the best possible care for patients in our overloaded public systems. And this paper suggests there is great potential.

Of course there is potential. There is no doubt that computers and robots will someday outperform docs. But to me (an experienced anesthesiologist), this day is probably futher off than most believe. Especially the robot part.

Guess the point of this sub-threat is, that as long as you cannot show it in a well-done study, the potential is just that: potential. But not something tangible.

The big problem is: what does a well-conducted study mean to people? For me personally, I won't be convinced until machines get the same info as the doc, meaning the whole messy file raw plus what can be heard, seen or touched directly, and that's it!

If a machine get's (statistically significant) better results than the docs with less information, that should be good as well, no?

Although I would tend to say we should expect machines to be significantly better than the average doctor. Before we role them out to everyone we should require them to be at least as good as the top 10% of doctors, I guess.

I.e. literally the first sentence if the top post.

I'm not saying doctor's expertise isn't valuable, but studies with even simple linear models have shown that it is much better to feed a doctor's expertise into an expert AI system than to try to teach the AI system to the doctor. And the weighting of the expertise relative to other factors is often lower than you might expect. Even a simple yes/no checklist can be invaluable if it is designed by the top minds of the field.

Clinical scoring (which is what I assume you are referring to as linear models) is mostly useful when the diagnosis is already more or less identified, to estimate prognosis and choose treatment options. They're no exploratory tools. When you 'feed the doc's expertise into a linear system', it means the expertise required to identify the problem in the first place has already done its job.

I don't have the original reference, it was an invited talk at UChicago, but something like https://www.sciencedirect.com/science/article/abs/pii/S09574... for example.

But regarding diagnosis, there are symptom checkers where you feed stuff in. I get the impression that multiple companies are in the process of building such tools based on crawling research papers to diagnose even very rare conditions. IIRC IBM Watson had such functionality.

NHS should be building this for PubMed and offering it for free.

One example: https://ada.com

I mean, it's not like rocket science. More like investing the effort to build a really big decision tree.

I don't think we want the doctor doing that though, that is where a lot of bias and inaccuracy creeps into the system. It'd be better if they used a checklist and evidence based process that can be engineered and improved over time.

If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit.

> I don't think we want the doctor doing that

You don't want your doc to give your problem his/her best shot !?

> used a checklist and evidence based process that can be engineered and improved over time.

Enticing and logically evident, isn't it? Well, except in practice it would cost so much to do that, that it's clearly not economically doable. And I'm not even sure it's technically doable in 2024.

> If there is some evidence they are gathering that ChatGPT doesn't have access to, an effort should be made to capture and systematise it. But there probably isn't very much of that for a typical doctors visit.

GPT is missing most of the info, because it's not equipped with 5 human senses and a brain to aggregate them. And that's, by far the most important info. You can see the challenge in solving this problem at a large scale, I'm sure.

> You don't want your doc to give your problem his/her best shot !?

I don't think the doctors gut feel is going to be their best shot. I want them using logic, evidence and reason. The medical industry has a long history of people slowly managing to convince doctors to use evidence instead of making bad guesses. Explicit processes generally lead to far superior outcomes.

Bluntly put, you are wrong regarding today's medicine. In 2024, medicine is a heterogeneous mix of art and science. Even for experts, separating real science from what others would want you to believe is difficult, and experience is essential in most cases to bridge the gaps in knowledge. The best summary I can formulate is 'in 2024, a good doc is a stellar educated guesser relying on lots of inexact knowledge, a substantial amount of experience, and a sprinkle of logical reasoning in the rare but critical steps where reasoning is applicable'.

That has been true from the dawn of medicine. And from the dawn of medicine, improvements have come from doing less art and more medicine; removing things like doctor's gut feelings from the process and replacing it with evidence based processes.

You're not wrong about what the present is, but you are misrepresenting what a desirable future looks like. We want to engineer the guesswork out of the system altogether.

We misunderstood each other, because I'm all for engineering. What I'm saying is that it's mostly not doable right now, because of insufficient information retrieval capabilities.

Reply on news.ycombinator.com