ChatGPT is really weird. I've gotten it to tell me that it can't write stories or poetry or anything like that.
I then asked it to write a haiku about its inability to write and it did a pretty good job.
I then asked it to write a haiku about its inability to write and it did a pretty good job.
Then I asked for a poem about blue chickens and it was basically the same poem.
It makes me think it could also be intentionally misled.