Hacker News Reader

Comment by nayroclade | Hacker News Reader

I have some good (or bad?) news for you then. Bing chat has no access to a Stockfish or other chess API. There is strong evidence that LLMs can learn and play games, including Chess. See this paper Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task by Kenneth Li et al.: https://arxiv.org/abs/2210.13382 or a more accessible write-up here: https://thegradient.pub/othello/

View on HN

One paper, making one claim, supported only by empirical results, in the current climate (where thousands of papers with similar claims are posted on preprint servers every month) is not "strong evidence". It's more like people saying stuff on social media.

> Bing chat has no access to a Stockfish or other chess API.

is there any evidence to back this claim?

It's very unlikely. The moves it makes are not always good. Any public facing chess engine API or the API they chose would have been at least 'good'.

if that's the case then I agree it is unlikely, but then I would posit it is highly unlikely that it flawlessly follows the rules of chess. why is this interesting then? of course an LLM can create the appearance of "knowing how to play chess" that passes basic scrutiny, given that chess notation is a thing and all, but that's totally separate from having "learned" an ironclad "understanding" of the rules of a relatively basic turn-based board game.

if Microsoft wanted to tout that its "AI" could actually play chess, then the obvious thing to do would be to hook in a chess AI system as I suggested, as opposed to somehow engineering (or prompt "engineering") a better-able-to-follow-the-rules-of-chess language model. again, this is how nearly everything is going to be going forward in this space: creating a product that has the appearances of being able to do so many different things, but in reality is many different specific subsystems wrapped in a nice language model package.

I'm getting really tired of seeing all these "look at what I made the 'AI' do!!" posts that really aren't interesting at all when you get past the headline.

It's interesting because it does demonstrate an understanding of chess, just like othello gpt demonstrated an understanding of othello.

It follows the rules of chess and makes legal rules the vast majority of times. Making illegal moves occasionally doesn't mean you don't understand chess. Not if you learned chess in a self supervisory fashion. It's also a vast improvement over the chess performance of cGPT which makes legal moves on basically random chance and flies off the handle quickly.

It's also just plain interesting that a language model can pick up how to play these games from training on text in the manner they do. If you don't find the idea that LLMs are universal computation engines interesting then i don't know what to tell you.

Reply on news.ycombinator.com