I worked for one of a few "game playing AI as a service" startups for several years, and we did in fact leverage the strong programming skills of LLMs to play games. When we pivoted from training directly on human gameplay data to simply having the LLM write modular code to play the game, we managed to solve many long-standing problems in our demo levels. Even on unseen games this works well, since LLMs already have a good understanding of how video games work generally.