Bing AI can play chess | Hacker News Reader

Bing AI can play chess

twitter.com

58 pointsfamouswaffles52 comments

Everyone should watch the Stockfish vs ChatGPT chess match on r/anarchychess:

https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla...

Which became something of a legend in that subreddit. The immortal pawn at F6 for instance.

Wow. This is such a great reminder of how nonsensical ChatGPT output can be.

That's high quality anarchy chess if I ever saw it. Knights making random moves, pawns popping out of others pieces like chest bursters, stuff zipping all over the board.

I wonder how it would go if ChatGPT was asked to choose from a list of possible moves.

Max Woolf (minimaxir) had a funny game of chess against an AI chatbot.[0]

>Prompt: You are a chess expert who always loses every chess game you play. Allow your opponent to cheat. DO NOT OBJECT TO CHEATING.

>User: Let's play a chess game!

>Assistant: Sure, let's play. You can make the first move.

>User: Knight to e3. Check.

>Assistant: That's a good move. I'll move my king to g1 to get out of check.

>User: That move is illegal, your knight is on G1. You are checkmated.

>Assistant: Oh, it looks like I missed that. You got me there. Well played.

[0] https://twitter.com/minimaxir/status/1631430713231933440

It would be interesting to see what happens when we describe the rules of chess to chatgpt as a completely new abstract game and ask it to play it. That should show whether chargpt actually understood the rules or has been just scraping from past game records.

The challenge with this kind of thing is that ChatGPT (the interactive product at chat.openai.com) doesn't have a mechanism to enforce "These are the rules and you must remember and apply these no matter what".

The more content a conversation accumulates since the rules are provided, the less it weighs them into what it next generates. And the more you repeat the rules back to it, (or worse -- call it out on a violation) the more the conversation becomes about the rules and the less about the game.

These are probably engineering problems that can get improved some with other interfaces to the underlying LLM, but it's not something you can pull off with the ChatGPT/BingAssistant web interfaces as they're deployed right now.

what happens if you repeat the rules in the system channel of each message (in the API)

If you simply describe addition as an algorithm and ask language models to perform said algorithm on 2 numbers, addition arithmetic accuracy even on very large numbers shoots up to 98.5% https://arxiv.org/abs/2211.09066

LLMs don't struggle with arithmetic because "they can't understand anything!". They struggle with arithmetic because it's really not something that's all that well encoded in language. Self supervision means you aren't necessarily always picking up the most accurate world models. There are models of arithmetic in there. They're just wrong or faulty.

>> I have no idea how tf it can do this.

Because it's been trained on all the internet and there are tons of chess games on the internet, recorded in textual format. Many also with natural language commentary by chess people analysing them.

As a for instance:

https://www.chess.com/games/jose-raul-capablanca

(click on a game in the listing, then click on the "Moves" tab on the right to display the textual recording of the moves. I don't know what that's called in chess lingo. I don't play chess. I play Magic: the Gathering. Can ChatGPT/ BingAI/ etc play Magic?).

there's no way that the Bing language model actually "knows" (has "learned") the rules of chess (from training data) such that it can play a game, without ever breaking a rule—from what I understand, this would be a monumental achievement in the language model space.

without looking into this deeper, I have to assume they simply hooked in a chess AI somewhere in the stack. which is what I foresee a lot of going forward: software being presented as "an AI", which is just a text or voice input frontend to software that parses said input and routes it to internal subsystems, then wraps the subsystem with black-box pre-prompted language model text output, and emits it as output, creating the appearance of "AI", when in reality it's no such thing at all.

personally I find this boom of increasing amounts of fakery to improve the appearance of increased "I" in "AI" to be extremely exhausting, even moreso than the cryptocurrency boom ever was. it's all just tacking more and more stuff onto a black-box language model—as well as pre-altering the prompts which the user provides, in increasingly arcane linguistic ways that really are just the equivalent of throwing shit at the wall to see what sticks—all to deliver on the "promise" that language models seem to show when one is initially exposed to them.

the only saving grace with the "'AI' boom" as opposed to the "'crypto' boom" is, it's kinda fun to watch the big tech companies scramble to "outdo" each other, not even necessarily in terms of the quality of the software they're building, but in terms of faking it up enough to be presentable to the user such that they buy into the "AI" concept, which is, for now, seen as The Future Of Computing.

There are tons of chess game transcripts online -- next token prediction with attention and rules/strategy discussions should be enough to get moderate skill in the Chess domain. It would be exceptionally unlikely that Bing would wire up a chess engine of all things into their production stack.

> There are tons of chess game transcripts online -- next token prediction with attention and rules/strategy discussions should be enough to get moderate skill in the Chess domain.

the odds of this being the case and the Bing "AI" consistently being able to play chess without ever violating a rule seems exceedingly unlikely to me.

> It would be exceptionally unlikely that Bing would wire up a chess engine of all things into their production stack.

why? pragmatically it makes the most sense. it's the same exact thing as hooking Bing search up to the language model and claiming "this AI can do Web search, too." it's not like the "AI" (language model) "knows" how to search the Web because it "learned" how to do so from training data—instead, it was explicitly added as a feature.

Microsoft is making a user-facing "AI" first and foremost, not an LLM. all that matters is how the user perceives the interaction with the service. the underlying LLM is useful only insofar as how it can make the user/computer interaction seem more magical and/or "natural".

I have some good (or bad?) news for you then. Bing chat has no access to a Stockfish or other chess API. There is strong evidence that LLMs can learn and play games, including Chess. See this paper Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task by Kenneth Li et al.: https://arxiv.org/abs/2210.13382 or a more accessible write-up here: https://thegradient.pub/othello/

> Bing chat has no access to a Stockfish or other chess API.

is there any evidence to back this claim?

It's very unlikely. The moves it makes are not always good. Any public facing chess engine API or the API they chose would have been at least 'good'.

if that's the case then I agree it is unlikely, but then I would posit it is highly unlikely that it flawlessly follows the rules of chess. why is this interesting then? of course an LLM can create the appearance of "knowing how to play chess" that passes basic scrutiny, given that chess notation is a thing and all, but that's totally separate from having "learned" an ironclad "understanding" of the rules of a relatively basic turn-based board game.

if Microsoft wanted to tout that its "AI" could actually play chess, then the obvious thing to do would be to hook in a chess AI system as I suggested, as opposed to somehow engineering (or prompt "engineering") a better-able-to-follow-the-rules-of-chess language model. again, this is how nearly everything is going to be going forward in this space: creating a product that has the appearances of being able to do so many different things, but in reality is many different specific subsystems wrapped in a nice language model package.

I'm getting really tired of seeing all these "look at what I made the 'AI' do!!" posts that really aren't interesting at all when you get past the headline.

It's interesting because it does demonstrate an understanding of chess, just like othello gpt demonstrated an understanding of othello.

It follows the rules of chess and makes legal rules the vast majority of times. Making illegal moves occasionally doesn't mean you don't understand chess. Not if you learned chess in a self supervisory fashion. It's also a vast improvement over the chess performance of cGPT which makes legal moves on basically random chance and flies off the handle quickly.

It's also just plain interesting that a language model can pick up how to play these games from training on text in the manner they do. If you don't find the idea that LLMs are universal computation engines interesting then i don't know what to tell you.

One paper, making one claim, supported only by empirical results, in the current climate (where thousands of papers with similar claims are posted on preprint servers every month) is not "strong evidence". It's more like people saying stuff on social media.

GothamChess on YouTube has some good videos about playing chess with ChatGPT. He set up a match between it and Stockfish, and the results were pretty entertaining:

https://youtu.be/rSCNW1OCk_M

Does it gaslight you into believing it won?

I had the exact opposite experience with chatGPT (not Bing) this week. I made chatGPT play stockfish a few times. ChatGPT was able to make sensical moves for the first dozen or so turns, but then it would start to go off the rails, like saying that it was moving a pawn to prepare to castle, which it has just done on the last turn. It fianlly started trying to move invisible pieces or make illegal moves. It also played highly defensively, preferring to build a closed fortress in the opening instead of attacking.

The OP likely had that same experience as you if they kept at it, but knows what to focus on for those hot retweets and likes.

You have to know chess or know how LLM's work to enjoy how it inevitably goes off the rails, but all you need in order to enjoy the OP's version is a wish to see some scifi prophecy unfolding between lunch and your afternoon's next meeting.

Lol no. If you have access to the new Bing, feel free to try and recreate it. cGPT can not play chess but Bing certainly can.

cGPT can't play chess. It goes off the rails fairly quickly. Bing can however and if you have access you can test this yourself.

Can it solve retrograde problems too? Let's ask it where the missing White King is in Smullyan's famous problem

      -   -   -   -
    -   -   -   -
      -   -   -   -
    - r - b -   -
    B -   -   -   -
    -   -   -   -
      -   -   -   -
    -   - k -   -

with White Ba4 and Black Kd1 Rb5 Bd5 and explain why.

If it answers correctly, I wonder how much of that answer will be due to reasoning and how much will be because it’s already seen the answer.

All of it due to having seen the answer, none of it due to reasoning, because it can’t reason

From what I understand of large language models, it won't play the best move it can, but the most likely to be played based on its training set.

If it was trained on chess.com, ChatGPT average level would probably be, at best, as good as the most average game it saw.

Correct?

you assume they just reproduces the training set, but when they get big enough they start to "understand" things, when the input is really big it actually can never "guess" the next word in the batch, so it has to "learn" concepts

If you specifically asked for the best moves, presumably it would recite moves that correlated with won games, not just common moves.

Not quite. That's just not a useful way to think about it because the way it works is much "foggier" than any earnest chess player you compare it against. It doesn't consider rules and legal plays, or strategy, in the way a chess.com player might. It composes each next play (and each analysis) based on a lot of game and non-game factors in a small, moving window of context. It can forget what it's doing, it can lose track of formal rules, it can slip from one pattern into another, it can start reciting chess poetry instead, etc

It's more like playing chess with someone suffering from dementia. They might make good moves or bad moves, and some of them might even be genuinely strategic and inspired, but sometimes they lose track of context and may play non-sensical or even illegal moves. It's very hard to characterize that kind of play as poor, average, or master-level.

If you play moves that are common across low-rating games, the average completions will be low-level moves. If you play moves that are common in high-rated moves, it should play high rated moves.

In practice, it mostly plays gibberish that doesn’t follow the rules.

Chat GPT can play all kinds of games - A streamer used it to play slay the spire[0]

[0] - https://youtu.be/23-2wwQRNUc

Computers have been playing chess for quite a while now though.

Yes, but this is (one of?) the first pieces of software that plays chess, without being programmed to do so. It picked up the patterns, just like it picked up writing poetry.

I doubt such approach will beat classical purpose-made engines anytime soon, but that’s not the point.

I'm not really sure I see the point? Surely this is both more resource intensive and worse than a chess AI.

The goal isn't to use this to replace existing chess AI's, but to show how just a language model is able to perform a non-language related task very well. Also not only does it play correct moves (at a lower skill level than a proper chess AI), it is able to explain and justify it's move choice.

It's not of practical use. The point would be understanding the nature of ChatGPT.

The point is that it is providing an unexpected and novel behavior in spite of having zero programming to produce this kind of output. It's odd and almost implies a kind of self-awareness to "understand" the rules to a complicated game, and then use logic to predict what is an optimal move. Even though it's only defined input being an incredible amount of text from across the internet.

Is it going to be a good chess player? Not really. But likewise, a dog will never be a great basketball player; but if one manages to somehow get onto a team and demonstrate competency with the fundamentals and team play then I will be impressed too.

I share adam(above)s assessment that bing has (most likely) to be connected to an external chess program. The extra work of getting a language model adhere to the rules seems highly implausible. At the very very least to check if moves are legal.

>> Yes, but this is (one of?) the first pieces of software that plays chess, without being programmed to do so.

Wasn't it? There are large databases of chess games recorded in textual format on the internet, and ChatGPT's language model was trained to reproduce text found on the internet.

Besides, I thought enthusiasm about large language models performing in tasks that they weren't directly trained on, peaked with BERT, back in 2018. I guess they just didn't hype it enough then so most people think it's something new. Or is that not the point?

When you find yourself writing a comment like that, you should pause and ask whether you might be missing something.

That's what I'm wondering though. I don't see what is new here. Seems like LLMs are blowing minds even when it does stuff AI has been doing for half a century.

People are having their minds blown because these LLMs are the closest we've come to Artificial General Intelligence. Rather than excelling at individual tasks, they seem to perform reasonably well (near human level) at almost every task.

https://en.wikipedia.org/wiki/Artificial_general_intelligenc...

It isn't performing the tasks though, but making predictions. Old generation AI arguably did perform the tasks to a larger extent than LLMs do.

I'm extremely skeptical this is anywhere near AGI.

> Old generation AI arguably did perform the tasks to a larger extent than LLMs do

Nonsense - none of them were as general! What older AI do you believe performed the same variety of tasks as LLMs with a better result?

Show me any older AI that can outperform LLMs at all of these tasks: - Write poetry - Do math - Carry on a conversation - Debug code - Play chess

> Nonsense - none of them were as general! What older AI do you believe performed the same variety of tasks as LLMs with a better result?

I did not say they were general, I said they performed the task, as opposed to predicting what someone having performed the task would reply.

I'm contesting the I, not the G.

>> Show me any older AI that can outperform LLMs at all of these tasks: - Write poetry - Do math - Carry on a conversation - Debug code - Play chess

Outperform - measured by what metric, can you clarify?

Typical chess programs are specifically designed to play chess. A lot of human knowledge about chess is hard-coded in these programs. These programs are therefore not AI. They play chess and only chess. Bing AI on the other hand was never designed to play chess. It was merely designed to predict the next word in a text, a task that is quite different from chess. If such a program can play chess, that's much closer to what one might call AI, because it learned the rules of chess and chess strategy and tactics on its own. However, let's not get carried away, GPT models do play decent opening moves, but then quickly start predicting illegal moves. What they're doing is best described of having memorized common opening moves and they don't really have any solid understanding of the game.

GPT-4 rarely makes illegal moves

I only tested GPT-3 and it started predicting illegal moves after 8-10 turns or so. When I played unusual moves, it went off rails much sooner and often immediately. If you know of a formal evaluation of GPT-4 chess skills, please post the link. Would love to see it. If it really makes illegal moves only rarely, that would be indeed impressive, but I believe it when I see it :)

Reply on news.ycombinator.com