Yep. I asked it to summarize a URL. It happily invented something completely fabricated and convincing sounding (until you go to the URL) based of the domain name and article slug.
Setting aside the creativity aspect (ie where do brand new ideas come from), which is a minuscule fraction of human thought, I suspect we are good pattern matching machines with the ability of extrapolation.
I think with sufficient data that computers can understand, this is eminently replicable.
These misanthropist slogans run up to contradictions so easily.
At the time it _seemed_ like it wasn't just doing a mindless string replace, because different parts of the url changed on most of the erroneous links. I figured it wasn't going out and spidering the link I gave, but it led me to believe maybe there was _some_ form of index that it simply hadn't prioritized prior, but was able to do with the additional context.
The training data however can be a few years old.
I have noticed this when trying to get it to generate some code, it was clearly using considerably old versions.
I have noticed the same thing when trying to use junior devs to generate some code, usually correlating with how old the most popular Stack Overflow question on the subject is.