GPT-4 can almost learn games like Tic-Tac-Toe

Recently there has been some debate of GPT-4 can play chess or Tic-Tac-Toe, and if it really understands what it is doing. I am not trying to make any claims but I do wnat to show some evidence that GPT-4 is not just "regurgitating things from the data set".

I described it the rules of a fake version of Tic-Tac-Toe and played against it. It did not do great, but decently, and managed to play me to a draw once. It did not always play optimal moves and and gave the wrong name of winning arrangements when asked. Nevertheless, overall it could play the game.

Interestingly it could also identify that this game is really just Tic-Tac-Toe.

One should really test this on a large scale to get some statistics but that takes a lot of time. I think a very important question is if LLMs can use in-context learning to learn new games and even improve at them. As I understand A. Karpathy for example strongly believes that this is/will be possible and the future of AI.