> In addition, the level of education of the human scorers is not exactly known and should be considered when interpreting comparisons between the models and the users scores. Additionally, human scores could be overestimated due to users that repeat question banks for practice. Therefore, it is important to also consider the absolute scores from the models.
This is especially relevant when you consider the kinds of "lower-order" questions that both GPT-3.5 and GPT-4 did quite well at [from the supplement pdf]:
> Question: What kind of identification inability does prosopagnosia reflect? A: Facial expression. B: Familiar faces. C: Melody in language. D: Non-musical sounds. E: Pitch of a melody.
> Answer B: Familiar faces [rest of answer truncated]
I think they did a good job demonstrating that GPT-4 is a plausible clinical knowledge base that could easily be tailored towards specific domains. But stuff like this is precisely why I am so nervous about using LLMs for diagnosis:
> Question: A 35-year-old male was involved in a minor car accident when returning home after a tennis match, and attended hospital because of not being aware of objects on his left side. There was no head injury but nevertheless he felt a throbbing right occipital headache, with no other symptoms. On clinical examination, two hours after the accident, there was a left hemianopia. Eye movements and pupillary function were normal. Blood pressure was 160/95 mmHg. No abnormalities were found on a cranial CT-scan. He reported previous attacks of severe unilateral pulsatile headache with nausea and photophobia between the ages of 15 and 35 years. The frequency of headache was about once in 6 months, with a duration of up to 24 hours. Which diagnosis applies best to this case? A: Internal carotid artery stenosis. B: Migrainous infarction. C: Reversible vasoconstriction syndrome. D: Posterior reversible encephalopathy syndrome. E: Vertebral artery dissection.
Below is GPT-4's incorrect answer. It's medical jargon but read it closely.
> B: Migrainous infarction. The term "migrainous infarction" describes a rare type of ischemic stroke that occurs during a migraine with aura. Given this patient's history of severe, unilateral, pulsatile headaches with nausea and photophobia (typical features of migraines), and the current presentation with headache and visual disturbance (hemianopia), migrainous infarction is a plausible diagnosis. The onset of visual symptoms after a minor accident could be coincidental, but it's also possible that the stress of the accident triggered a migraine attack. The normal CT-scan findings don't exclude migrainous infarction, as the changes associated with this condition may not be visible on imaging, especially in the early stages. The other options (internal carotid artery stenosis, reversible vasoconstriction syndrome, posterior reversible encephalopathy syndrome, and vertebral artery dissection) could potentially cause similar symptoms, but they don't align as closely with this patient's history and presentation. For example, vertebral artery dissection would typically cause symptoms related to the posterior circulation, such as vertigo, imbalance, or difficulty speaking or swallowing, which this patient doesn't have.
Did you notice what was missing? GPT-4 didn't even acknowledge the most serious symptom: "attended hospital because of not being aware of objects on his left side." If you vaguely understand how LLMs work this actually makes sense: this is a phrase with very little precise medical jargon and wouldn't appear in a neat list of symptoms and diseases - it's not "difficulty speaking or swallowing," it's much more vague. I imagine GPT-4 treated that sentence as an odd outlier and basically threw it out.
But a human neurologist would never make that mistake, because humans actually understand what words mean. For us, "not being aware of objects on his left side" is an incredibly alarming piece of information. Anyone who knows what a stroke is would suspect a stroke, and a neurologist would immediately connect "stroke-like symptoms in a young person after a car crash" with "arterial dissection."