You mean combine sort of related information from all over the internet into new statistically generated phrases that appear to answer the question, but still contain all kinds of hallucinations, fabrications, and wrong statements?

Then yes, they do.

Of course they will, they’ll answer whatever you ask. How do you know if what they’re saying is true?