Preparing your game for deterministic netcode
yal.cc
yal.cc
I took the gamble and picked Unity DOTS and NetCode for our current MOBA-esque game (not other packages, we bolt regular Unity visuals on much like the V in MVVM). Outside the headaches of using preview tech, it has been amazing what we have been able to do compared to regular OOP hell.
By not having hidden game state lurking in classes, lambdas or coroutines, we have been able to hide much of the networking layer as an encapsulated set of glue systems that shuttle just enough data across the wire that the view-layer can present the game to the user. Movement and other prediction has been much easier too, because the game state is just data and not intertwined with behavior, I just run most of the same algorithms locally as I do on the server — AI, steering, etc.
This technique still requires a lot of foresight and challenging architecture decisions of course. I forced our core gameplay logic to not know anything about the visuals, which has made some things harder. However when it came time to make a headless build with no graphics, it was as simple as flipping a switch. No changes in our gameplay code.
I have a principal engineer from Sea of Thieves on my team, and he has also breathed huge sighs of relief working on this project. Unreal is the complete antithesis to this design pattern, and is full of hidden state, magic behavior, and heisenbugs.
Mine? Don’t. By that I mean, and assuming you haven’t already, spend a few years making small games from start to finish. Pick games to copy, don’t even hold yourself to having a unique spin on things. The goal being to 1) practice for the real thing and 2) decide if you actually want to make games or just fantasize about doing so. The latter isn’t a judgement, I’ve almost left games twice because of the emotional burnout.
For a point of reference, I spent two years getting up at 6am twice a week with a friend to gamejam and practice. All in prep to go indie. This was even after 6 years making games and leading two.
The result? We almost failed completely. If I hadn’t decided to say fuck it and stick the game on PS4 for fun, we would have grossed a grand total of $100k for a year of work. The more depressing fact? $100k is considered a significant success for an indie game on Steam.
Happy to provide more anecdotal experience around this if you like.
In addition to what hesdeadjim said, I'd also like to add don't blindly accept the common gamedev maxims. For example, "Make the game YOU want to play" is repeated as good advice, but I haven't found it to be very useful.
First off, if I'm working as the sole developer and have total control over design decisions, it's easier to follow that advice than if I'm working with multiple people who all play different genres of games. A lot of the time people will only use gamedev-jargon to back up their design decisions (immersion, positive feedback, vertical slices, etc), which are hard to argue about because they're so subjective and ill-defined.
Also, keep in mind how many times Level 1 gets tested. At a certain point, no matter how excited I am about the game, I'm going to get sick of playing it. From that point on, I'm no longer developing a game I want to play. I'm developing a game I should want to play but don't (I think this adds a lot to burnout). Maybe start the project as a game you want to play, but make peace with the fact that it's ultimately not for you to enjoy playing. It's for the players. You're very likely going to be sick of the game by the time it ships.
"Make it juicy" and "avoid premature optimizations" are more maxims with limited use, and are even contradictory in a lot of instances. For juice/polish: Focus on the core gameplay mechanics. Once those are set in stone, make them juicy if you want/have the time. A juicy boring game is still a boring game. For performance/optimization: If a core part of your game is having 10,000 enemies on the screen at once, you're going to have to make design decisions based on performance at the very beginning of development. Kinda related: I've shipped a game where at the beginning of development, I suggested using Multiple Render Targets (we were doing deferred rendering with the XNA Framework) to speed up the rendering (instead of rendering everything, switching render targets, rendering everything again, switching render targets, repeat...) but the lead dev took the "avoid premature optimizations" maxim to heart, and we ended up with a significantly hobbled final product on PS4s and lower-end PCs. I guess I could have gone into "crunch mode" and rewrote the entire rendering pipeline to make up for poor decisions at the start of the project, but I wasn't willing to do that.
It was my dream to make games from the age of probably 13 to ~28. I'm 35 now, and am not all that interested in games any more. I enjoy doing "gamedev" as a hobby, because I've realized my preferred method of "playing" is to program little toys I can tinker with for a few hours and then abandon. I also have MDD, so YMMV. Best of luck, though.
I got a job as a game developer, and I wrote a game engine. I was so burnt out and unhappy that I had to go on anti-depressants.
Years later, some of the most fun I had was optimising financial reporting databases. Why? I dunno, it was just more fun somehow. It wasn't my dream, and I felt like I was that kid that wanted to become an astronaut and ended up in middle management at some megacorp. But you know what? Optimising queries uses the same skills as optimising a game engine! It's fun, and you get paid for it. You get immediate feedback from users, instead of complaining from entitled brats on forums two years later...
I'd say if you want to not hate you life building any software, start with the data/schema/et. al. Before you open Visual Studio, open Excel and conference in your business stakeholders for a few design sessions.
We recently started down a "data-driven" path with our B2B application. We went from ~90% of the application being code-based to ~95% of the application being configuration-driven in a matter of months. We can bootstrap an entire customer's setup by doing some simple JSON copypasta now.
The best general model I have found so far would be to declare a global Domain type, and then put a List<T>/T of each constituent domain type within it. No nesting of complex types (reference types) should be permitted. All properties in the domain model types can map cleanly to/from SQL tables. The overall schema should be in 3NF/BCNF/DKNF. Relation types will be required. You can trivially serialize instances of the Domain type for whatever purpose. Each serialized copy of the Domain type can be thought of as a deterministic snapshot of all state, assuming you did everything correctly elsewhere.
Use of language features such as LINQ can make this approach almost fun to work with. A clean, well-managed schema is the most powerful tool in your box when dealing with complex problem domains.
This read badly. Why is JSON in the picture?
In situations where we have "bad things" happen, we can simply dump the entire domain instance to disk and then reload it on a dev machine.
The 'desync' stuff was surprisingly easy to resolve. Instead of trying to make all game state deterministic (a near impossible task in some languages, where for example even basic math functions don't have fully defined rounding charactistics), I went for another approach...
Detect desyncs by periodically taking a ram snapshot of the whole game and calculating a CRC over it. It means your game state needs to be serializable, but you probably need that anyway so a new player can join the game mid game.
If the CRC doesn't match, do a majority vote to find out which player is wrong, then do the rsync algorithm to sync differences between the players. In nearly all cases there will only be a few bytes of differences.
There is no need for the game to halt or lag during any of this - the game can continue as normal. Players are likely seeing the exact same thing on the screen, because the differences between game states are probably an error of 0.00001 in some player position.
When the corrected state has arrived, replay frames to bring it up to the current timestamp and switch out states. Job done. User sees no glitchyness. Developers don't have to worry about tiny bits of nondeterministic behaviour in libraries out of their control. Everyone's happy.
Make sure it doesn't happen every frame though, or you'll waste a lot of CPU.
Doing a CRC of full game state also poses some additional requirements - e.g. any pointers must be relative offsets and any dynamic allocation should be deterministic and within the same block. It does, however, also make it easier to implement serialization/deserialization, as mentioned by Jyaif in another comment here.
I implemented netcode for a canvas game I made a while ago over websockets. It used client-server with various RPCs from/to the clients and worked pretty well. I’m just starting on a kind of follow up of that game that will have more discrete movement so it was useful to read this overview. I’d maybe like to add that often games make a distinction between “authorized” actions players take (eg spell casts that damage other players), that often use the client-server RPC mode, and the “p2p rollback” actions that are less strict but generate more traffic like player movement. I believe World of Warcraft uses this model, and I also worked on a FPS that IIRC also worked this way: a gun fire was RPC over TCP; movement was done via UDP broadcast.
In particular, he talks a lot more about problems in floating point determinism, which was missing from the posted article IMO. It is possible to make it work without resorting to fixed point, however it is a big pain point.
netplayjs: https://github.com/rameshvarun/netplayjs
Instead of serializing individual objects, you serialize all the memory relevant to the gameplay by using a custom memory allocator whenever the gameplay related code runs. This custom memory allocator is designed to allow taking fast memory snapshots, and supports restoring memory snapshots.
This allows rollbacking any code, including in my case rollbacking user-submitted Lua scripts.
I wrote about it at https://www.jfgeyelin.com/2021/02/a-general-state-rollback-t...
(aside: code blocks in the post could use a fallback font in case Menlo isn't installed)
Edit: You can P2P the game state if you want, you just have to make sure it's usable for those with poorer internet connections (since you are sending/receiving much more data, position/velocity vectors etc. as opposed to just key presses, per number of players).
This seems very costly just to hopefully get faster prediction (for those clients that are close together, and whose NAT is not too broken for P2P to work). That information would also not be authoritative, since the simulation on the far-away central server is still the authority, so I'm not even sure how much it would actually help.
The CPU cost is largely there anyways, as you are updating everything on the client from the server anyways. Any games that rely more heavily on physics will have more work to do to simulate yes, although basic gravity/velocity calculations with some linear interpolations thrown in is very lightweight, even when talking about 100 players. Bandwidth is quite small, how much data does it take to send a single key stroke? A number of bytes really, multiply by 100 players, multiply by say 2 or 3 keystrokes at the same time, multiply by 60 frames per second, and you still get a very reasonable (dare I say negligible?) amount of bandwidth.
The point is not for it to be authoritative whatsoever, the point is to not have to wait on two round trips from the server to get location data from your opponents, but try to do it in one round trip. If your connection to your opponent is lower latency/more stable if done through the server, then that should be the fallback of course. Movement prediction/smoothing happens even to your own personal character, even on a good, low ping connection to the server, the good games just know how to hide it effectively, but you often still get people complaining about hit registration. There is no perfect model outside of everyone having amazing internet connections.
But in real game development, there's too few situations where lockstep fits, as it makes the game pretty slow to react to player's actions. And rollback brings so much complexity to the whole game that it significantly impacts development costs.
I built a small game with Quantum a couple years back when I took a “sabbatical” (shipped an indie game didn’t know what to do next). I found it extremely limiting at the time, and despite determinism being the promise, I ran into constant desync issues.
Things have probably changed in the last few years, but Photon isn’t a large company and they have limited resources to improve it. YMMV
I was well on the way to implement a "Drive Mode" when I came across several hurdles relating to the net code.
First I was on the path of implementing a pure FPS like game mode where any unit could be controlled, be it tanks, cyborgs, turrets, towers, bunkers, artillery, airplanes, boats, even walls! (Although they obviously couldnt move)
The first prototypes were awesome, really inspiring and the community gave lots of brainstorming.
We would be making battle royales, one on ones, elite single missions, awesome multiplayer coops, you name it.
Then, I found out that the netcode only allows for move orders, go forward, turn left, speed 100, etc. Seemed manageable. So far so good.
But wait. What about aircraft? Ouch, thats important though. Now they just move with WSAD, and you just stop midair.
After that, I found out we couldnt simply click somewhere to shoot. Netcode only allowed to fire with intent to hit a specific unit. Hmmm ... Ok .. so turret cant rotate except when attacking.
And, dont click to shoot, but "click to attack".
I was initially really enthusiastic over reinstating this awesome game mode, but these restrictions totally killed it unfortunately.
I guess the bottom line is that yes robust deterministic netcode is good and all, but it kinda locks in your game to what it currently is designed to do and fundamental changes to that will sometimes be overwhelmingly difficult if not impossible.
PR on Github: https://github.com/Warzone2100/warzone2100/pull/829
My initial prototype, no relation to the netcode, point to shoot, camera follows turret: https://youtu.be/itsd29NebGE
My second prototype, no camera following aim, click to attack (iirc): https://youtu.be/ODBmOakkqWk
> 8 frames in 16ms
You have other articles discussing client/server netcode. The one from Quake 3 is a good start (and it is open source).