I really thought the author was going to start writing chess notation and Copilot would actually play a game, that would have been impressive.
I gave it the input
> Let's play chess! I'll go first.
> 1.e4 c5
Here's the first 7 turns of the game it generated https://lichess.org/bzaWuFNg
I think this is a normal Sicilian opening?
At turn 8 it starts not generating full turns anymore.
Update: I tried to play a full game against a Level 1 Stockfish bot vs GitHub Copilot. It needed a bit of help sometimes since it generated invalid moves but here's the whole game
It resigned after it got stuck in a long loop of moving it's queen back and forth.
4) C3 is just a terrible move that gives away a knight for nothing.
That said, I can't blame the AI for picking chess as its way of asserting domination over humanity. "No human has beaten a computer in a chess tournament in 15 years."
http://dbacl.sourceforge.net/spam_chess-1.html.
https://www.newswise.com/articles/ai-chess-engine-sacrifices...
They mention that they manage to expand the window in terms of characters greatly (to 4096?), which I take as implying they recomputed the BPEs on the source corpus to make it much more tailored and relevant to source code (which makes sense, because I would expect source code to be far more verbose and repetitive, in terms of syntax and vocab, than Internet-wide natural language, and so simply using the old GPT-2/GPT-3 BPEs won't work well).
Which is fantastic if you're doing Python, and that wide window is part of how they do tricks like being able to put in parts of debugging sessions, but if you are going to use Copilot for non-source-code generation, I wonder what the consequences might be...? BPE issues can be hard to notice even when you are looking for them and have the vocab at hand to check.