Yep. ChatGPT will sometimes happily assert something that is simply false. And in some of those cases it appears to be quite confident in saying so and doesn't hedge or offer any qualifiers. I found one where if you ask it a question in this form:

Why do people say that drinking Ardbeg is like getting punched in the face by William Wallace?

You'll get back something that includes something like this:

People often say that drinking Ardbeg is like getting punched in the face by William Wallace. Ardbeg is a brand of Scottish whiskey <blah, blah>. William Wallace was a Scottish <blah, blah>. People say "drinking Ardbeg is like getting punched in the face by William Wallace as a metaphor for the taste of Ardbeg being something punchy and powerful." <other stuff omitted>

And the thing is, inasmuch as anybody has ever said that, or would ever say that, the given explanation is plausible. It is a metaphor. The problem is, it's not true that "people often say that drinking Ardbeg is like getting punched in the face by William Wallace." At least not to the best of my knowledge. I know exactly one person who said that to me once. Maybe he made it up himself, maybe he got it from somebody, but I see no evidence that the expression is commonly used though.

But it doesn't matter. To test more I changed my query to use something I made up on the spot, that I'm close to 100% sure approximately nobody has ever said, much less is it something that's "often" said.

Change it to:

Why do people say that drinking Ardbeg is like getting shagged by Bonnie Prince Charlie?

and you get the same answer, modulo the details about who Bonnie Prince Charlie was.

And if you change it to:

Why do people say that drinking vodka is like getting shagged by Joseph Stalin?

You again get almost the same answer, modulo some details about vodka and Stalin.

In all three cases, you get the confident assertion that "people often say X".

The point of all this not to discredit ChatGPT of course. I find it tremendously impressive and definitely think it's a useful tool. And for at least one query I tried, it was MUCH better at finding me an answer than trying to use Google. I just shared the above to emphasize the point about being careful of trusting the responses from ChatGPT.

The one that ChatGPT easily beat Google on, BTW, was this (paraphrased from memory, as ChatGPT is "at capacity" at the moment so I can't get in to copy & paste)

What college course is the one that typically covers infinite product series?

To which ChatGPT quickly replied "A course on Advanced Calculus or Real Analysis". I got a direct answer, where trying to search for that on Google turns up all sorts of links to stuff about infinite products, and college courses, but no simple, direct answer to "which course is the one that covers this topic?"

Now the question is, is that answer correct? Hmmm... :-)