GPT-4V(ision) Unsuitable for Clinical Care and Education: An Evaluation

Not in the human nature. A friend of mine just asked me how to make ChatGPT create a PowerPoint presentation. He meant the pptx file. It can't. Googling for him I learned that of course it can create the text and a little more surpringly even the VBA program that creates the slides. That's our of scope for that friend of mine. He was very surprised. He was like, with all it can do why not the slides?

Because the average user of AI is not a Hacker News user who understands its limitations, and the ones who do understand their limitations tend to exaggerate and overhype it to make people think it can do anything. The only real fix for it is for companies like OpenAI to encourage better usage (e.g. tutorials), but there's no incentive for them to do so yet.

I wrote a rant a few months back about the greatest threat to generative AI is people using it poorly: https://minimaxir.com/2023/10/ai-sturgeons-law/

Same reason we can't trust people to drive 35 MPH when the road is straight and wide, no matter how many signs are posted to declare the speed limit. It's just too tempting and easy to become complacent.

That, and these companies have a substantial financial interest in pushing the omniscience/omnipotence narrative. OpenAI trying to encourage responsible AI usage is like Phillip Morris trying to encourage responsible tobacco use. Fundamental conflict of interest.

I spoke w/ Marvin Minsky once back in the 1990s and he told me that he thought "emergent properties" were bunk.

As for the future I am certain LLMs will become more efficient in terms of resource consumption and easier to train, but I am not so certain that they're going to fundamentally solve the problems that LLMs have now.

Try to train one to tell you what kind of shoes a person is wearing in an image and it will likely "short circuit" and conclude that a person with fancy clothes is wearing fancy shoes (true much more often than not) even if you can't see their shoes at all. (Is a person wearing a sports jersey and holding a basketball on a basketball court necessarily wearing sneakers?) This is one of those cases where bias looks like it is giving better performance but show a system like that a basketball player wearing combat boots and it will look stupid. So much of the apparent high performance LLMs comes out of this bias and I'm not sure the problem can really be fixed.

Because none of us really know what we mean by "intelligent".

When we see a thing with coherent writing about any subject we're not experts in, even when we notice the huge and dramatic "wet pavements cause rain" level flaws when it's writing about our own speciality, we forget all those examples of flaws the moment the page changes and we revert once more to thinking it is a font of wisdom.

We've been doing this with newspapers for a century or two before Michael Crichton coined the Gell-Mann Amnesia effect.

There are production ready AI tools in hospitals, with FDA approval. The writers of this article just decided to try a non-FDA approved tool and ignore the approved ones

There are lots of papers about using some kind of LLM to cut costs and staffing in hospitals. Big companies believe there is a lot of money to be made here, despite the obvious dangers.

A quick Google search found this paper, here's a quote:

"Our evaluation shows that GPT-4V excels in understanding medical images and is able to generate high-quality radiology reports and effectively answer questions about medical images. Meanwhile, it is found that its performance for medical visual grounding needs to be substantially improved. In addition, we observe the discrepancy between the evaluation outcome from quantitative analysis and that from human evaluation. This discrepancy suggests the limitations of conventional metrics in assessing the performance of large language models like GPT-4V and the necessity of developing new metrics for automatic quantitative analysis."

https://arxiv.org/html/2310.20381v5

For sure there's some waffling at the end, but many people will come away with the feeling that this is something GPT-4V can do.

because everybody is shit scared of the implications of it being good enough to replace them. This is just a protective defense mechanism because it threatens the status quo upon which entire careers have been built.

"It is difficult to get a man to understand something when his salary depends on his not understanding it."

It's called Artificial Intelligence.

Because companies like OpenAI market the shit out of them to get people to believe that ChatGPT can do anything.

Two reasons:

1. Because you've got one or more of the below spinning it into either a butterfly to chase or a product to buy:

- 'Research Groups' e.x. Gartner

- Startups with an 'AI' product

- Startups that add an 'AI' feature

- OpenAI [0]

2. I'm currently working on a theory that a reasonable portion of population in certain circles is viewing ChatGPT and it's ilk as the perfect way to mask their long COVID symptoms and thus embracing blindly. [1]

[0] - The level of hyuperis in some articles about ChatGPT3 reminded me a little too much of the fake viral news around the launch of Pokemon Go, adjusted for fake viral news producers improving quality of tactics. Especially because it flares up when -they- do things... but others?

[1] - Whoever needs to read this probably won't, but; I know when you had ChatGPT wrote the JIRA requirements and more importantly I know when you didn't sanity check what it spit out.

This is super dangerous. It's WebMD all over again but much worse. It's hard enough to diagnose but now you have to fight against some phantom model that the patient thinks is smarter than their doctor.

After reading the diagnosis I caught myself wanting to examine the MRI to see if the bright area really exists, which means I fell for this too. Imagine being the person who received this diagnosis. Of course you're going to be concerned, even if it is LLM garbage.

There is probably a disproportionate amount of texts describing the presence of a tumor on a scan than texts describing what a normal scan would look like. So this is expected. But I think that is not the purpose of a LLM. There should be a classification algorithm detecting the presence or not of a tumor and then a a LLM can be used to generate a description of it or why the scan is normal.

And I don't know any, but there is probably already tools to help radiologists, what they call their "own AI".

I'm using chatgpt right now to create a Hugo page.

For whatever reason Hugo's docu is weird to get into while chatgpt is shockingly good in telling me what I actually look for

Definitely. Saw a video recently mentioning the increase in well-paid gigs for therapists in a metro area, which ask only to record all therapist-patient interactions and treat it as IP. It seems likely that the data would be part of a corpus to train specialists models for psychotherapy AI, and if this kind of product can actually work I don’t see why every other analytical profession wouldn’t be targeted and well underway. Lots of guesses there though, and personally I hope we aren’t rushing into this.

Instead of comparing against some "average doctor", they used a few doctors as "source of truth"

> All images were evaluated by two senior surgical residents (K.R.A, H.S.) and a board-certified internal medicine physician (A.T.). ECGs and clinical photos of dermatologic conditions were additionally evaluated by a board-certified cardiac electrophysiologist (A.H.) and dermatologist (A.C.), respectively

Yeah, I don’t think we’re more than a decade off from GPs being replaceable with AI. But you can bet they will lobby hard against it.

IMHO, we need structural change in the way US healthcare works. Slathering some AI on top won't solve our cost problems but it will negatively impact our already declining quality of patient care.