Agreed that is one of the hairier challenges a concept like this faces. I do think that argument is worth unpacking though. Would a render node in your neighborhood have acceptable latency? IIRC they have demoed "real-time" NFT's based on an Unreal Engine node streaming frames at acceptable e2e latency.

Personally I'd think a bit broader than frame-streaming for achieving "console in the room" levels of video game fidelity.