I think there's something to this. I have a theory that LLMs are implicitly trained primarily to impress people, since that's what motivates those who work on them, excites the general public, and convinces conferences to publish papers.

I don’t think so. In Wordle you have to guess the word in six attempts. It’s a fun game and often simple.

So it could be that ChatGPT picked up on a pattern in the training data where after a couple of guesses, a lot of the time people pick the right word.

So statistically it might go like. Guess a word. Probably not the right one. Guess a couple more and suddenly it’s statistically likely to be the correct word, and because of that the LLM ends up outputting the congrats and so on