I can hardly believe this claim, anyone who has played some amount of DOOM before should notice the viewport and textures not "feeling right", or the usually static objects moving slightly.
The entire thing would probably crash and burn if you did something just slightly unusual compared to the training data, too. People talking about 'generated' games often seem to fantasize about an AI that will make up new outcomes for players that go off the beaten path, but a large part of the fun of real games is figuring out what you can do within the predetermined constraints set by the game's code. (Pen-and-paper RPGs are highly open-ended, but even a Game Master needs to sometimes protects the players from themselves; whereas the current generation of AI is famously incapable of saying no.)
I suspect there is a reason for this: running while turning doesn't work properly and makes it very obvious that the system doesn't have a consistent internal 3D view of the world. I'm already getting motion sickness from the inconsistencies in straight-line movement, I can't imagine turning is any better.
That's literally how the human rating was setup if you read the paper.
What I'd posit is that it's not actually a very good replication of the game but very good a replicating short clips that almost look like the game and the short time horizons are deliberately chosen because the authors know the model lacks coherence beyond that.
Do you mean the PSNR and LPIPS metrics used in paper?
Off the top of my head DOOM is open source so it should be reasonable to setup repeatable scenarios and use some frames from the game to create a starting scenario for the simulation that is the same. Then the input from the player of the game could be used to drive the simulated version. You could go further and instrument events occurring in the game for direct comparison to the simulation. I’d be interested in setting a baseline for playtime of the level in question and using sessions of around that length as an ultimate test.
There are some on obvious mechanical deficiencies seen in the videos they’ve published. One that really stood out to me was the damage taken when in the radioactive slime. So I don’t think the analysis would need to particularly deep to find differences.