Activations inside Llama-2 show an internal model of the world | Hacker News Reader

Activations inside Llama-2 show an internal model of the world

twitter.com

16 pointsmindwok8 comments

I remember reading something similar about describing a chess game to one of these large language models and being able to find an internal representation of the chessboard in the parameters. Or something like that. My memory is a bit foggy, but there’s definitely other examples out there of large language models building a pretty comprehensive world model internally. That’s the best way to accurately predict the next token.

A really interesting article!

https://thegradient.pub/othello/

"How do these models achieve this kind of performance? Do they merely memorize training data and reread it out loud, or are they picking up the rules of English grammar and the syntax of C language? Are they building something like an internal world model—an understandable model of the process producing the sequences?

"From various philosophical [1] and mathematical [2] perspectives, some researchers argue that it is fundamentally impossible for models trained with guess-the-next-word to learn the “meanings'' of language and their performance is merely the result of memorizing “surface statistics”, i.e., a long list of correlations that do not reflect a causal model of the process generating the sequence. Without knowing if this is the case, it becomes difficult to align the model to human values and purge spurious correlations picked up by the model [3,4]. This issue is of practical concern since relying on spurious correlations may lead to problems on out-of-distribution data.

"The goal of our paper [5] is to explore this question in a carefully controlled setting. As we will discuss, we find interesting evidence that simple sequence prediction can lead to the formation of a world model. ..."

I'm no expert on LLMs, but I've always found the "it's just predicting the next word in a sentence" trope to be a huge misrepresentation of what LLMs are actually doing. To me this is a really cool example of how LLMs actually learn more than just simple word prediction and start to exhibit real intelligence for some tasks.

Intelligence, or consciousness? As a novice in all the technical aspects, I fail to see the distinction between a 'self updating internal model', and a human conscious experience.

I think some of the current characterizations of LLMs is due to unwillingness to posit algorithms as entities.

Is a PID loop conscious then?

I would love to see the code to verify this is not a confirmation bias.

Can someone explain a bit more how it works? I don't understand the "We then train linear probes on the last token activations to predict the real latitude and longitudes of each place." I've looked up linear probing on Wikipedia, but it's not obvious how it can be used to predict lat and long.

https://arxiv.org/abs/2310.02207

Reply on news.ycombinator.com