Hacker News Reader

Comment by marginalia_nu | Hacker News Reader

Computers have been playing chess for quite a while now though.

Yes, but this is (one of?) the first pieces of software that plays chess, without being programmed to do so. It picked up the patterns, just like it picked up writing poetry.

I doubt such approach will beat classical purpose-made engines anytime soon, but that’s not the point.

I'm not really sure I see the point? Surely this is both more resource intensive and worse than a chess AI.

The goal isn't to use this to replace existing chess AI's, but to show how just a language model is able to perform a non-language related task very well. Also not only does it play correct moves (at a lower skill level than a proper chess AI), it is able to explain and justify it's move choice.

It's not of practical use. The point would be understanding the nature of ChatGPT.

The point is that it is providing an unexpected and novel behavior in spite of having zero programming to produce this kind of output. It's odd and almost implies a kind of self-awareness to "understand" the rules to a complicated game, and then use logic to predict what is an optimal move. Even though it's only defined input being an incredible amount of text from across the internet.

Is it going to be a good chess player? Not really. But likewise, a dog will never be a great basketball player; but if one manages to somehow get onto a team and demonstrate competency with the fundamentals and team play then I will be impressed too.

I share adam(above)s assessment that bing has (most likely) to be connected to an external chess program. The extra work of getting a language model adhere to the rules seems highly implausible. At the very very least to check if moves are legal.

>> Yes, but this is (one of?) the first pieces of software that plays chess, without being programmed to do so.

Wasn't it? There are large databases of chess games recorded in textual format on the internet, and ChatGPT's language model was trained to reproduce text found on the internet.

Besides, I thought enthusiasm about large language models performing in tasks that they weren't directly trained on, peaked with BERT, back in 2018. I guess they just didn't hype it enough then so most people think it's something new. Or is that not the point?

When you find yourself writing a comment like that, you should pause and ask whether you might be missing something.

That's what I'm wondering though. I don't see what is new here. Seems like LLMs are blowing minds even when it does stuff AI has been doing for half a century.

People are having their minds blown because these LLMs are the closest we've come to Artificial General Intelligence. Rather than excelling at individual tasks, they seem to perform reasonably well (near human level) at almost every task.

https://en.wikipedia.org/wiki/Artificial_general_intelligenc...

It isn't performing the tasks though, but making predictions. Old generation AI arguably did perform the tasks to a larger extent than LLMs do.

I'm extremely skeptical this is anywhere near AGI.

> Old generation AI arguably did perform the tasks to a larger extent than LLMs do

Nonsense - none of them were as general! What older AI do you believe performed the same variety of tasks as LLMs with a better result?

Show me any older AI that can outperform LLMs at all of these tasks: - Write poetry - Do math - Carry on a conversation - Debug code - Play chess

> Nonsense - none of them were as general! What older AI do you believe performed the same variety of tasks as LLMs with a better result?

I did not say they were general, I said they performed the task, as opposed to predicting what someone having performed the task would reply.

I'm contesting the I, not the G.

>> Show me any older AI that can outperform LLMs at all of these tasks: - Write poetry - Do math - Carry on a conversation - Debug code - Play chess

Outperform - measured by what metric, can you clarify?

Typical chess programs are specifically designed to play chess. A lot of human knowledge about chess is hard-coded in these programs. These programs are therefore not AI. They play chess and only chess. Bing AI on the other hand was never designed to play chess. It was merely designed to predict the next word in a text, a task that is quite different from chess. If such a program can play chess, that's much closer to what one might call AI, because it learned the rules of chess and chess strategy and tactics on its own. However, let's not get carried away, GPT models do play decent opening moves, but then quickly start predicting illegal moves. What they're doing is best described of having memorized common opening moves and they don't really have any solid understanding of the game.

GPT-4 rarely makes illegal moves

I only tested GPT-3 and it started predicting illegal moves after 8-10 turns or so. When I played unusual moves, it went off rails much sooner and often immediately. If you know of a formal evaluation of GPT-4 chess skills, please post the link. Would love to see it. If it really makes illegal moves only rarely, that would be indeed impressive, but I believe it when I see it :)

Reply on news.ycombinator.com