Comment by ADeerAppeared | Hacker News Reader

Comment by ADeerAppeared | Hacker News Reader

Parent Full thread

I should clarify, "of the problem given" refers to the problem given in a prompt.

As you note, transformer (and indeed, most ML) models do create a "world model". They're useful for 'specific' intelligence tasks.

The problem for general tasks lies in their inability to create specific models. To stick with the board game example: The model can't handle differently shaped boards, or changes to the rules.

I could ask a human and a chess-trained AI system to, for a given chess board state and piece, what places that piece can move to. Both have their model of chess.

But if I then ask, "With the rule change that the pawn can always move two spaces", the AI cannot update their model. Where for the human this would be trivial. The human can substitue in new logic rules, the AI cannot.

And that is very core of what's required for generalized logic and "thinking" in the way most tasks require it. What's so troublesome about current generative AI is that it's trained to be extremely general (within the domain of text generation), so their internal models aren't all that good.

Ask an LLM the chess problem above and you might even get a good answer out, but it doesn't generalize to all such chess problems, especially not more complex ones.

The paper on Othello is of course a very limited model, useful because it's simple enough to study and complex enough to have interesting behaviour.

But the general takeaway is that this is evidence that large transformers like GPT, which are trained to predict text, are fully capable of developing emergent models of parts of that input space whenever it is convenient for minimising the loss function. In practice this means that GPT may have internal models of the semantics of human dialogue that are sophisticated enough for it to get by in the enormous variety of prediction tasks we throw at it.

I agree with you that it's likely these internal models aren't very detailed (for the reason you wrote - they're very general). The linked blog actually talks about this at the end - an OthelloGPT trained to be good at Othello rather than just able to play legal moves ends up with a worse board model. Presumably because it needs to "invest" more in playing better moves. But if you agree with the blog's take then this is just a matter of scale and training. And whether it's possible or not for them to develop models capable of complex tasks like strategy games with shifting rules is certainly not something you (or anyone else for that matter) can say with certainty right now.

Edit: I should clarify we're using "model" in two senses here. There's the actual transformer model, but what I and the blog are talking about is specific weights and neurons _inside_ these transformers that learn to predict complex features of the input space (like legal moves and board updates in the case of OthelloGPT). These develop spontaneously during the training process, which is why they are so interesting. And why they are not really analogous to the "ML models" you refer to in your first two paragraphs.

If you're going to suggest something you think an LLM can't do I think at the very least as a show of good faith you should try it out. I've lost count of the number of times people have told me LLMs can't do shit that they very evidently can.

I explicitly say that LLMs could do it in my response. As a show of good faith you should try reading the entire comment.

Yes, I'm using simple examples to demonstrate a particular difference, because using "real" examples makes getting the point across a lot harder.

You're also just wrong. I did in fact test, and both GPT 3.5 Turbo and 4o failed. Not only with the rule change, but with the mere task of providing possible moves. I only included the admission that they may succeed as a matter of due diligence, in that I cannot conclusively rule out they can't get the right answer because of the randomization and API-specific pre-prompting involved.

> "For chess board r1bk3r/p2pBpNp/n4n2/1p1NP2P/6P1/3P4/P1P1K3/q5b1 (FEN notation), what are the available moves for pawn B5"

I did read your entire comment, and that is what prompted my response, because from my perspective your entire premise was based on LLMs failing at simple examples, and yet despite admitting you thought there was a chance an LLM would succeed at your example, it didn't seem you'd bothered to check.

The argument you are making is based on the fact that the example is simple. If the example were not simple, you would not be able to use it to dismiss LLMs.

I am not surprised that GPT 3.5 and 4o failed, they are both terrible models. GPT4-o is multimodal, but it is far buggier than gpt-4. I tried with claude 3.5 sonnet and it got it first try. It also was able to compute the moves when told the rule change.

Reply on news.ycombinator.com