> If you ask ChatGPT how to use it, it will roleplay a character called “Assistant” from a counterfactual world where “how do I use Assistant?” has a single, well-defined answer. Because it is role-playing – improvising – it will not always give you the same answer. And none of the answers are true, about the real world. They’re about the fantasy world, where the fantasy app called “Assistant” really exists.

"Role playing" is a great analogy for what these models are doing. If an AI is asked a reasonable question it doesn't know the answer to, it won't go "I'm sorry, I'm not sure, let me go check," it performs improv by generating something that kinda feels like what the right answer might be