Their ability to generate facts is a consequence of the word statistics they are truly using. I think it's fair to say the statistics explanation is a more accurate interpretation.
The responses of the model don't represent what it understands per some internal model, because there is no "it", only models of the sources it was trained on, and it'll just as happily generate lies as truths, or smart vs dumb answers (it's all just words) if that is what its source modelling calls for.
What most people mean when they say the model is hallucinating/bullshitting isn't where it has learnt a lie, but rather where is is operating "out of distribution", and is therefore (unknowingly) generating a mashup from multiple only loosely related/matching source contexts.
The fancy part in the paper is figuring out how to perturb the intermediate layers in the way you want. But the findings are not impressive.
Note also that the "probe geometry" stuff is so speculative they left it out of the academic paper completely.
In the same way, it has been known since the 90s that if you take the matrices from Finite Element Models and visualize them as graphs, structures appear that kind of resemble the physical appearance of the object being modelled. Here for instance is for a helicopter:
http://yifanhu.net/GALLERY/GRAPHS/GIF_SMALL/Pothen@commanche...
Yet nobody thinks Finite Element Models have an internal mental representation of the world.
At this point, I'm not sure some wouldn't argue that.
The difference is, put the AI on a loop, with constant feedback, learning. Instead of just a 'pre-trained' model. Make the actual model, live, always learning, so the context window is infinite. This of course would not be for everyone, because it would take all the resources of the training infrastructure to be focused on one person/view. But that gets closer to the human mind, and at that point, we probably couldn't say for sure that the 'perturbations' aren't experiencing something subjective.
Where is the proof that humans have an internal mental representation of the world.
Building a world model for a perfect information game is different than building an mental model of the external world.
In context learning is a well known property of LLMs, while real world generalization, often described through the common sense problem is not.
To me, 'mental models' are personal, internal representations of external reality, which LLMs currently lack, being limited to the corpus.
- Construct arbitrary text that isn't just grammatically but semantically coherent
- Derive intent, subtle intent, from user queries and responses
- Emulate endless different personalities and their reactions to endless stimuli
- Describe in detail the statics and dynamics of the world, including sight, smell, touch and sound
do not have a model of the external world? What do you think a "corpus" means in this context? How is the "corpus" of sensory and evolutionary data that makes you up in any way different?
LLMs are excellent common sense reasoners, and they generalize just fine. Why exactly do you think they get things _subtly_ wrong? Make up API syntax that looks sensible but isn't actually implemented? In order to make these guesses they need to have generalized, they need an understanding of the structure underlying naming, such that they can produce _sensible_ output even if they lack the hard facts.
But just few months ago, saw example of AI, from video, building an internal representation of the world. An internal model of the world. Everyone saying this can't be done, it already is. Maybe can argue it wasn't an LLM, and then I'd say were nitpicking over which technology can do it or not. We already have example of tying them together, symbols and LLM's.
Might be related. https://www.nature.com/articles/d41586-024-00288-1 https://www.technologyreview.com/2019/04/08/103223/two-rival...
Perhaps this is what you're looking for (similar technique, larger model?) https://www.anthropic.com/news/golden-gate-claude
> In the “mind” of Claude, we found millions of concepts that activate when the model reads relevant text or sees relevant images, which we call “features”.
> One of those was the concept of the Golden Gate Bridge. We found that there’s a specific combination of neurons in Claude’s neural network that activates when it encounters a mention (or a picture) of this most famous San Francisco landmark.
Sounds like a mental model to me! An internal representation of an external concept which exists in the real world.
I don't really know if using an "experiment" is the right way to test this?
Like, if you want to test such hypothesis you should first prove that these LLMs were created with that specific function in mind, we are not observing a natural phenomena, it is a manmade(manmade...loosly) algorithm .
You don't know the internal architecture of the system and you are testing in a reality narrow situation, this is basically the scientific method applied incorrectly.