What percentage of patients have blood clots in their lungs and a history of lupus, like the article described? That's not on the same level as a common cold at all.
> In one case in the Harvard study, a patient presented with a blood clot to the lungs and worsening symptoms.
That's a single anecdotal fluke from the study, which is misleadingly used to represent the headlining percentages.
If you read the linked paper, it says the LLMs did not outperform any group of doctors in the most important cases:
> The median proportion of cannot-miss diagnoses included for o1-preview was 0.92 [interquartile range (IQR) 0.62 to 1.0], although this was not significantly higher than GPT-4, attending physicians, or residents.
And again, the bigger issue is that skimming nurse's notes and predicting the next tokens, as the study made the doctors do, is not how doctors diagnose medical conditions.
The lupus anecdote on its own is irrelevant to the whether the statistics are being interpreted in valid ways or not. Also, I said nothing about luck.