Hacker News Reader

Comment by psb217 | Hacker News Reader

The othello paper is annoying and oversold. Yes, the representations in a model M trained to predict y (the set of possible next moves) conditioned on x (the full sequence of prior moves) will contain as much information about y as there is in x. That this information is present in M's internal representations says nothing about whether M has a world model. Eg, we could train a decoder to look just at x (not at the representations in M) and predict whatever bits of info we claim indicate presence of a world model in M when we predict the bits from M's internal representations. Does this mean the raw data x has a world model? I guess you could extend your definition of having a world model to say that any data produced by some system contains a model of that system, but then having a world model means nothing.

View on HN

Well I actually read Neel Nanda's writings on it which acknowledge weaknesses and potential gaps. Because I'm not qualified to judge it myself.

But that's hardly the point. The question is whether or not "general intelligence" is an emergent property from stupider processes, and my view is "Yes, almost certainly, isn't that the most likely explanation for our own intelligence?" If it is, and we keep seeing LLMs building more robust approximations of real world models, it's pretty insane to say "No, there is without doubt a wall we're going to hit. It's invisible but I know it's there."

My point was mainly that this claim: "we keep seeing LLMs building more robust approximations of real world models" is hard to evaluate without a well-formed definition of what it means to have a world model. Eg, a more restrictive definition of having a world model might include the ability to adapt reasoning to account for changes in the modeled world. Eg, an LLM with a proper model of chess by this definition would be able to quickly adapt to account for a rule change like "rooks and bishops can't move more than 4 squares at a time".

I don't think there are any major walls either, but I think there are at least a few more plateaus we'll hit and spend time wandering around before finding the right direction for continued progress. Meanwhile, businesses/society/etc can work to catch up with the rapid progress made on the way to the current plateau.

I think we're largely in agreement then, actually. I'm seeing "world models" as a spectrum. World models aren't even consistent among adult humans. I claim LLMs are moving up that ladder, and whether or not they've crosses a threshold into "real" world models I do not actually claim to know. Of course I also agree that it's very possible, maybe even likely, that LLMs aren't able to cross that threshold.

> this claim ... is hard to evaluate without a well-formed definition of what it means to have a world model

Absolutely yes, but that only makes it more imperative that we're analyzing things critically, rigorously, and honestly. Again you and I may be on the same side here. Mainly my point was that asserting the intrinsic non-intelligence of LLMs is a very bad take, as it's not supported by evidence and, if anything, it contradicts some (admittedly very difficult to parse) evidence we do have that LLMs might be able to develop a general capability for constructing mental models of the world.

Reply on news.ycombinator.com