I've done some experiments with a little camera-equipped robot navigating by prompting a LMM with stills and producing captions that include military-style o'clock directions. Some promising results with a weekend of tinkering. I'm sure one could play Doom that way, too.