But many chatgpt hallucinations are catastrophically wrong. For example when querying for things related to advanced scientific subjects in physics and medicine for starters from my own experience.

I really want to give this a pass but the worst thing is that you can't really see where it turns from correct to incorrect, like you would usually pick up with a human teacher or peer when you start to get a feeling they really don't know what they're talking about. ChatGPT will gladly keep hallucinating references and reasonings that sound superficially plausible until you look them up, because that's what it's trained to output..

If they could just find a way to cut it off when it starts to become too unsure, it would be a big improvement.