Comment by TacticalCoder | Hacker News Reader

Comment by TacticalCoder | Hacker News Reader

My worst "impossible" bug I encountered in 1991 (and I should blog about it one day), while writing a DOS game (386/486 PCs era). Game was a mix of C and assembly and for the life of me I couldn't reason about it / recreate the conditions to make the bug appear. It was seemingly random, only very rarely happening.

After days I gave up and invented the thermonuclear weapon: I decided to rewrite the entire game's engine to be deterministic. This took me along while but then I could record events (joystick/keys direction/firing etc.) and at which frame these inputs happened, and could deterministically replay the whole game.

Thing is: back then deterministic games engine / replay based on inputs didn't exist yet (AFAIK). At least I didn't know of any.

The first time I remember reading about a fully deterministic game engine was on Gamasutra, a post-mortem on the first Age of Empire.

So basically my first impossible bug made me discover the idea of deterministic game engines and tiny "replay" save files.

The actual bug? Well eventually it appeared but now I had save files and, sure enough, I could have the whole game replayed automatically and the bug would now show up. And so I knew it was now just a matter of squashing it. Just some good old C dangling pointer IIRC. When the hero had the option to get two shots (usually he only had one) and when one of the shot shot was still active when the level was cleared, that shot would keep being alive in the next level, but invisible and would invariably lead to corrupting the memory of the next level.

Fun stuff.

For the DOS game Terra Nova (1996) I made our game engine fully deterministic. I guess this was a year before Age of Empires but well after you. It was amazing how much easier this made debugging. The effect was so great that I can't even put a multiplier on it, because it moved bugs from the "we'll never reproduce this" category to the "just see what happened and fix it" category. Some of these replays represented more than half an hour of gameplay too.

One thing that surprised me is that it also found a bunch of bugs waiting to happen (uninitalized variables / dangling pointer sort of stuff) that would trigger an error when replaying from a file didn't produce the same results as the original play (we had a checksum of game state that we could check).

That is really cool! There may have been others, earlier ones: just not that I knew of.

You came with the idea yourselves or you knew about other game developers doing that?

> Some of these replays represented more than half an hour of gameplay too.

Same... That was really my main motivation: sometimes needing to play for 20 minutes before the bug would show up.

> (we had a checksum of game state that we could check)

Ooooh I love that: that is plain bad---! So you not only had your deterministic engine, but a way to directly identify any discrepancy between the original state and the replayed one. I didn't think about that!

It's amazing that it let you identify bugs before they even stroke.

I don't remember hearing about other developers doing the same thing. I think it was just that I had a lot of experience debugging deterministic programs (like command-line tools) and it was infinitely more pleasant than trying to debug an interactive graphical program, so it was worth seeing whether we could make the game itself deterministic.

I've never made a system that ambitious again, but one thing I've learned from that experience is to never ever call global rand(); always always create your own RNGs that you can run explicitly, and if you have multiple systems that can be disabled independently (e.g., we were able to turn off graphics during our replays), give them each their own RNG.

Is there any downside to writing deterministic games? It seems like the only sane way to do it, but I guess it adds some complexity overhead for the initial write otherwise everyone would do it by default?

There aren't many significant downsides I know of, it's just really easy to accidentally make your game not deterministic.

You have to entirely isolate the game state from any sources of non-determinism. The latter can include: subtle CPU timing issues, GPU timing, other GPU artifacts, the system clock, timing from IO operations. If you want the state to be deterministic across machines (useful for debugging multiplayer stuff) then you also need to include floating point operations (some chips behave differently on some boundary cases, I think) as well as some graphics operations (thinks like texture operations and rounding are always bitwise identical across GPUs).

If any bit of non-determinism sneaks in from one of these, it will wander through and pollute any other operations and data that depend on it. Flushing out non-determinism bugs can almost feel as difficult as debugging a non-deterministic engine.

I was at EA when the Madden team refactored the engine to be deterministic. It took a full cycle of bug hunting, but it was marvelous once it got there. QA could just send over a replay file and any dev could simply load up the replay and repro the bugs. In fact, the user-facing replay system in the game ("Let's watch that play again in slow mo!") was simply restarting the engine and then replaying the user inputs deterministically to resimulate the whole game again.

There's a bunch of excellent blog posts from the Factorio developers about tracking down tiny bits of non-determinism (since it uses deterministic multiplayer, as there is way too much dynamic world state to update constantly over the network).

Yeah it’s just harder and potentially less performant. There are different levels—deterministic across the same architecture, different architectures etc…

The simplest example I can think of is being deterministic across different frame rates.

Imagine you move a player by adding x to it’s position each frame. You adjust x based on the frame rate so that you don’t move faster on a faster computer.

So on a computer tuning at 30 FPS you move 10 pixels each update. But on a computer running at 60 FPS you move 5.

You have walls that are 6 pixels wide. On the 60fps computer it works fine, but on the 30fps machine you can teleport through the walls.

I guess this is similar to the Fallout 76 "physics is tied to framerate" snafu (where looking at the ground makes you go faster because the fps goes up...)

https://gamerant.com/fallout-76-speed-hack/

I seem to recall '90 Stunts (also known as 4D Sports Driving) was somewhat deterministic as it featured replays, with which I recalled abusing the physics engine with carefully laid out tracks, and looping over the replays to much hilarity

> I should blog about it one day

Yes, please!

Reply on news.ycombinator.com