It is not about what the model tells you.

This paper shows an emergent world model in an LLM that was taught to play otello moves https://ar5iv.labs.arxiv.org/html/2210.13382

https://arxiv.org/pdf/2303.12712.pdf This paper discusses (among other things) how a GPT4 model navigated between rooms in a text adventure game and was able to create a map afterward. Literally building a model of the world as it was navigating and drawing a map of that afterwards

I mean, just like you can create 1-line python script that claims "I am an AGI" and have that fact be false, you can have ChatGPT tell you it has no explicit world model, while exhibiting behaviors that can only really be explained by it having some sort of model of the world inside it.

Fine-tuning is like a PR agent teaching someone what sorts of things not to mention on TV even though they may be true.