That's easy to test, invent a new chess variant and see how the model does.
We are pawns, hoping to be maybe a Rook to the King by endgame.
Some think we can promote our pawns to Queens to match.
Luckily, the Jester muses!
"Chess but white and black swap their knights" for example?
None of these changes are explained to the LLM, so if it can tell it's still chess, it must deduce this on its own.
Would any LLM be able to play at a decent level?
These LLM's just exhibited agency.
Swallow your pride.
>'thinking' vs 'just recombinating things
>By that standard (and it is a good standard), none of these "AI" things are doing any thinking
>"Does it generalize past the training data" has been a pre-registered goalpost since before the attention transformer architecture came on the scene.
Learning is thinking, reasoning, what have you.
Move goalposts, re-define words, it won't matter.
If the chess specialization was done through reinforcement learning, that's not going to transfer to your new variant, any more than access to Stockfish would help it.