i've experimented with this a little and wrote about the tradeoffs here https://themaximalist.com/infinityarcade/

it's currently pretty difficult to get the models to output sane game state, it should be possible soon though

in the meantime i'm calling this YOLO CODE!