You don't know what it's been training on and you don't have the resources to validate the output. It's LLM 101: what you shouldn't do
LLMs aren't an oracle of truth, science and medicine can't solve everything. It's OK to be sick and not have answers, if you've "done everything" and the experts don't have an answer chat gpt won't automagically summon an answer from its statistical soup. It sucks but it's a fact of life, we're not gods, and neither are llms.
I am smart. I am capable of understanding deeply technical subjects.
I have a team of smart doctors to present my findings to.
I’m looking for plausibility. Not the full diagnosis.