Super cool to see this idea working. I had a go at getting an LLM to play Pokémon in 2023, with openai vision. With only 100 expensive api calls a day, I shelved the project after putting together a quick POC and finding that the model struggled to see things or work out where the player was. I guess models are now better, but also looks like people are providing the model with information in addition to the game screen.
https://x.com/sidradcliffe/status/1722355983643525427?t=dYMk...