Hacker News Reader

Comment by chpatrick | Hacker News Reader

Good thing it's "incapable of reasoning"!

It is incapable of reasoning, actually - at least in this case. It has no internal understanding of chess which is why it makes illegal moves.

Mostly it didn't make illegal moves though, since illegal moves mean resignation and it won more than it lost.

Making 60 legal moves in a row in one game would be the coincidence of the century unless it had some knowledge of the rules of chess.

It's a probabilistic text model. If it has a 99% probability of generating an acceptable "next" thing to say, that means it would have a 50/50 chance of generating 60 legal moves in a row, which doesn't seem all that coincidental.

And the 99% probability isn't an evidence of understanding chess?

I don't know.

Part of me wants to say no, that the model "thinks" in terms of text it has seen and so knows from chess forums it has seen that certain text representing moves come naturally after previous moves' text. It doesn't understand anything other than certain text comes after other text.

But yeah at the same time I can see how it is thinking inside the world we built for it. We have senses like touch, smell, sight. The only "sense" these models have are an input text box. Would we even necessarily recognize intelligence when it is so different from our own?

So does it understand chess like I do? No, it cannot. Does it understand chess at all? I'm not sure. I'm not sure I'd understand chess in it's world either though.

Markov chains are probabilistic text models and rather far from 1400 elo

How did it win 11 out of 19 games then, blind luck?

raw statistical power.

a game of chess becomes « new » after a few moves. starting middlegame, you’re in unknown territories and have no statistics to refer to..

I'm referring to the statistical power of the model. For example, if you replace GPT4 with GPT2 it will lose every game, because the statistical power is lower. Increasing the statistical power doesn't make the model understand any better, it just makes it more likely to generate a response that aligns with human expectations.

"Statistical power" isn't some magic property of GPT4. It can produce statistically more likely moves because somewhere deep down it can model chess.

It isn't a model of chess, it's a model of internet text, if it was a model of chess it wouldn't make illegal moves.

if it didn't have at least some kind of model of chess, it wouldn't be able to play past midgame.

Simply because on a new position, moves from other positions aren't applicable at all.

How do you know that? It has billions of parameters, some of them may well be for internal understanding of chess?

What experiment would you run to determine if a given text input / text output interface had an "internal understanding of chess"?

What if you prompted with something like:

    Let's play a game chess.  Use the standard rules except that ....

Basically perturb the context to something a human would easily adapt to if they first knew the rules of chess but that would be difficult (or at least not obvious) to extrapolate from training data by ChatGPT (or more generally an LLM)

I think internal understanding requires internal processing.

According to this functional definition, the way we are currently using language models basically excludes understanding. We are asking them to dream up or brainstorm things – to tell us the first things they associate with the prompt.

Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. If that works, it would be one step closer to a true AGI that can be said to understand things.

There is a lot of confusion around the Chinese Room Argument. I think it makes a valid point by demonstrating that input/output behavior alone is insufficient for evaluating whether a system is intelligent and understands things. In order to do that, we need to see (or assume) the internal mechanism.

> Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts.

It can do that while it generates output. Humans do the same thing when they figure out what they really mean while they're trying to express it.

I was thinking more about the equivalent of a human noticing that the initial answer they were going to give is wrong and then thinking about the topic for a while before coming up with a better answer.

Is a normal chess program capable of reasoning?

In the narrow field of chess, yes.

I would say so. Logical reasoning is not a hard thing to program a computer to do: the really hard part is to reason about the real world, rather than just a manually-defined world model.

Traditional algorithms are vanguards of pure reason. Neural networks are super-intuition. Stockfish unites the two, executing an efficient intuitive heuristic search over the solution space of “moves in a chess game” – but no matter how cool the robot arm you build for it, Stockfish could never learn to assemble furniture.

The Monte Carlo analysis AlphaZero used functioned as a sort of multi-step reasoning for it. GPT can use its token buffer for some multi-step reasoning but that sort of interferes with providing a conversation with the user so it's much less effective.

Reply on news.ycombinator.com