Yeah, my best guess is they're probably including the previous N frames as context into generating the next model. This works to preserve continuity over a short amount of time (as you say, briefly looking at the ground and then back up), but only a short period of time.

For these kinds of models to be "playable" by humans (and, I'd argue, most fledgling AI agents), the world state needs to be encoded in the context, not just a visual representation of what the player most recently saw.