This is my exact problem with ChatGTP, it looks great when you don't know what it's talking about, but as soon as you do, it looks foolishly over-confident in it's answers which are very clearly wrong.
It's not a crystal ball, it's a mirror.
There is obviously more to human intelligence than text based conversation but it is pretty humbling that such an aspect of ourselves can be replicated and be so convincing at such an early stage and perform better than some humans even when it makes stuff up: toddlers can't talk, kids are smarter but don't have the technical knowledge, most adults only have a few specific areas of expertise, etc.
but not always; And that's what makes it scary. You know people will use it and trusting it.