A test that doesn't properly inform you someone/thing's abilities is a rubbish test. From their paper, you would think being able to pilot robots to handle chores or stack objects or whatever would be well beyond LLMs. But it's not.

https://tidybot.cs.princeton.edu/ https://innermonologue.github.io/

Anyone who wants LLMs to plan and is actually interested in teasing the extent of those abilities knows how to structure planning requests.

It's extra funny because humans can't actually generate plans the way he tests LLMs to either.

Also Seeing output from GPT that demonstrates intelligence, reasoning, or whatever, and saying it is not real reasoning/Intelligence etc, is like looking at a plane soar and saying that the plane is fake flying. And this isn't a nature versus artificial thing either. The origin point is entirely arbitrary.

You could just as easily move the origin to Bees and say, "oh, birds aren't really flying". You could move it to planes and say, "oh, helicopters aren't really flying." It's a very meaningless statement.

Internal processes are entirely irrelevant.