This is a case where LLMs and similar generative approaches are outright dangerous and damaging. Not so much immediately but in that they erode trust in machine assisted diagnosis or treatment and set back adoption of things that could actually work substantially.

Namely tools like expert systems which are uniquely suited for the application can easily be confused with the current batch of "AI" despite being much more sane and useful.