If you run a game entirely between colluding parties, cheating speed runners can just hack it to do whatever they want anyway. See the Dream Minecraft thing from several years back. Speed running claims may be cheated in a thousand ways. It's up to the people who care about it to establish and enforce rules.

But if you're running a multiplayer game with random elements and aren't colluding, you don't have to let a malicious party set the RNG seed to whatever they like just because you agree on it at game start. There's any number of simple cryptographic protocols that allow each peer to contribute equally to the RNG state based on having a separate commitment phase. And it's a lot easier to run a quick cryptographic setup than it is to have constant input-driven adjustment.