I’m not convinced that these are particularly useful tests of very large language models like GPT-4. These models have been trained on enormous amounts of text, including theory of mind tests. And the example scenario in the paper is very simple and likely related to things that the models in question have seen before by a simple substitution.

So I would actually be a bit surprised if the models failed this type of test regardless of whether they have any understanding of theory of mind.

Note that I’m making no claim about how intelligent or understanding of humans or neurotypical these models are. My claim is that it’s very hard to tell given how incredibly knowledgeable the models are.