Or: Almost a quarter of AI responses to healthcare queries wrong
> A total of 34 participants — comprising faculty, staff and undergraduate and graduate students — submitted 212 prompts and AI-generated responses to real and imaginary health concerns written from both patient and doctor perspectives. Participants were allowed to choose one of four LLMs to use for the contest: ChatGPT-4o, ChatGPT-3.5, Gemini-1.5 Pro and Llama3-8b.
This press release links to an arXiv article dated a year ago, which ran tests using AI models that were already seriously out of date at that time. The practical upshot of which is that, with respect to the question people care about, and which the headline claims to answer, this is basically pseudoscience.