I only tested GPT-3 and it started predicting illegal moves after 8-10 turns or so. When I played unusual moves, it went off rails much sooner and often immediately. If you know of a formal evaluation of GPT-4 chess skills, please post the link. Would love to see it. If it really makes illegal moves only rarely, that would be indeed impressive, but I believe it when I see it :)