Are you trying to use an actual 2D board representation in text to play? In my experience, GPT-3.5 specifically is pretty bad at tasks that require spatial reasoning, even very basic ones.
I've done it both ways, while GPT-4 seems to be better at sticking to valid moves, neither play well, and neither noticed when I won. If you claim victory at any point, they'll congratulate you and concede the game, which is pretty funny.