Hathora: Multiplayer Made Easy
blog.hathora.dev
blog.hathora.dev
I've been focusing on single player experiences because I haven't found that comprehensive guide that explains the nuts and bolts of multiplayer for video games. If I knew more then I could pull in more of my software engineering knowledge.
I fear that there is more incentive to create projects like this to hide that knowledge away from game developers than to thoroughly explain how to build one's own multiplayer solution.
I post this here in case someone knows of such a comprehensive guide for multiplayer.
The key of multiplayer is that you have to master streaming abstractions via a TCP OR datagram UDP. Ultimately, you can build your own transport, but I recommend starting with TCP.
At this point, the key on architecting your game depends on what kind of state you are synchronizing. There are many interesting ideas in the space (deterministic, shared clocks), but the really hard way is to invent your own and live with the consequences.
I highly recommend you write your own RPC layer because that will teach you many of the fundamentals of networking.
However, the transport for multiplayer is a fraction of the total problem. The next problem is server management. Sure, you can fire up a server on AWS, host some code, and then scale it up vertically.
If your games are small sessions, then you have the challenge of match making players to the hosts. This will include novel load balancing aspects due to the long lived nature of games. However, you further have the challenge of keeping those hosts alive in an environment that is trending towards "treat your servers like cattle".
If your games are large scale, then you have to figure out how to distribute state across multiple machines which entails a bunch of problems.
Regardless of scale, the next question is around availability. What happens when you process dies? We can treat host death as rare, but deployments will happen every so often.
This is a fun space to explore!
Try using TCP, but turn off delayed ACKs. That stupid fixed timer from the Telnet echo era is still in there.
The ACK delay timer is a fixed timeout, classically about 100ms. It was put in to piggyback ACKs of echoed characters on the echoed character packet, for use with TELNET. So it's a human-scaled delay. It's not a function of round trip time.
It's hard to turn off. There's the socket option TCP_QUICKACK, but you have to reset it on every "recv" call on Linux. For Windows, see SIO_TCP_SET_ACK_FREQUENCY.
For a multiplayer game, you usually control both client and server, so you can set this at both ends. It improves performance when set at the other end, so if you only have control of one end, it's not that helpful.
Anyway, try it, and do something like fast remote procedure calls to test.
The series doesn't really get in to doing things like rollback and the like, which would be "beyond fundamentals", as noted by the fact that a lot of AAA games don't even have good rollback net code.
Blizzard folks gave a GDC talk on the Overwatch architecture and their rollback implementation that is good for a conceptual discussion[2]
Here's a Gist w/ some rollback net code pseudocode that got posted in a gamedev subreddit once that I bookmarked but didn't read myself[3]
EDIT: Here's the lower-level game networking series from Gaffer as well, "Building a Game Network Protocol"[4]
[1]: https://gafferongames.com/categories/networked-physics/
[2]: https://www.youtube.com/watch?v=zrIY0eIyqmI
[3]: https://gist.github.com/rcmagic/f8d76bca32b5609e85ab156db383...
[4]: https://gafferongames.com/categories/building-a-game-network...
I used what I learned from reading the articles on their site to implement my own input prediction and rollback netcode for my personal side project. It's a pretty useful resource, and helps organize my project in other ways too such as informing me of ways I can structure event messages, handle input valuation, etc.
1. Follow these two guides on setting up a Node.js server on AWS:
https://hackernoon.com/tutorial-creating-and-managing-a-node...
https://hackernoon.com/tutorial-creating-and-managing-a-node...
2. Write some client-side Javascript functions to send and receive requests from your server.
3. Use Unity's browser scripting to call the Javascript functions:
https://docs.unity3d.com/Manual/webgl-interactingwithbrowser...
At this point, you'll be able to have clients send to the server and receive responses. You can then implement whatever multiplayer solutions you fancy.
Once you make it so that everything that happens is an atomic event, serializing them between client and server becomes more straightforward. As a bonus it also makes your engine pretty wonderful to code in. There are some downsides, and the newer ECS patterns might not be compatible, but you probably don't need ECS performance for game logic.
From there it's a matter of prediction and synchronization. No way you can wait for some varying 50-70ms for input to be reflected on your screen, and other player units need to also have their movement smoothed. Lets say you're developing an FPS and a player takes a shot at another who has almost managed to dive for cover, you have 3 states to rectify -- the shooter who thinks they're not behind cover yet, the server who thinks they're almost behind cover, and the player being shot who is behind cover. 3 steps, all 30-40ms apart, all technically "valid". So you have to develop algorithms to intelligently handle rectifying for game logic as well as smoothing the motion. For game logic it's usually taking both perspectives with a certain degree of leniency (yes shooter they would have been in that position on your screen when you'd fired the shot x milliseconds before your shot event reached our server). For motion you need the clients to run the games themselves for those frames between updates and let them take updates as suggestions where appropriate. This is where "rubber-banding" comes from, because the server (which maybe crashed or is dropping packets for more than a second) hasn't validated that you're in the spot you think you're in so the client keeps yanking you back.
A lot of multiplayer doesn't even need low ping, after all.
Check out gamasutra and GDC vault. Lots of good bits in there.
The most important thing IMO when developing a multiplayer game is that you do not let clients make decisions. This is such a painfully easy way to prevent one avenue of cheating, yet some FPS games (like PUBG) allow it (or at least, did allow it in the past) and so one way of cheating was for the client to tell the server it scored a hit despite the target not being line of sight. And I don't mean "the target was running behind a wall and latency allowed the hit", I mean "The target has been inside this house, not exposed to any windows, and still got hit" or "The target was on the other side of a 40-foot tall hill and couldn't possibly have gotten hit".
The second is that when possible, you don't send any state to clients that they don't need to see. This is harder in a real-time game like an FPS, but in, say, a Poker game, it's easy.
My implementation was pretty simple. The simulation ran at 60 ticks per second. Both client and server would store the last 120 tick states (The state of the game was pretty small, less than 1 kilobyte). Clients would send their user inputs and the tick number of the input. When the server got it, it would go back to that tick and re-simulate the game to the current tick assuming the user kept that input. It would also relay that input to other players, which would also roll back and re-simulate.
In this model, clients have zero authority. The only messages they can send are player inputs and pings. The simulation is deterministic, and I used TCP, so theoretically, the server would never have to send a full state.
To avoid players "jumping" on the screen when they change direction, you render their location from the previous tick. It makes their location 1 frame behind, but it smooths out latency correction.
To prevent players from using lag to cheat (ie, waiting to see what other players do, then send inputs with old tick numbers that would move them into an advantageous position), I would simply reject any inputs from more than 30 ticks (1/2 second) behind, send a message to the client that they're lagging, and send a full copy of the game state to the client to resync them so that the player sees what the server sees after their input was ignored.
Overall, it was pretty simple to implement, prevented cheating, and handled latency well enough.
The problem is, I got bored after writing all that code and never actually fleshed out the game mechanics.
There are far too many variables to say that server side calculation is the only real way to program.
I think the opposite is true, most good multiplayer networking uses this model, and deviations from it are the niche. You may even have large player base models that shard on top of using this design, but the original premise takes hold, and there are entire generations of young hobbyists who I think probably refuse to believe this in large part because most of them aren't doing sophisticated enough networking to say otherwise, and as soon as they even try, they'll be reinventing the wheel.
The nature of multiplayer games forces you to make architectural decisions that will always essentially look like a client-server, server authoritative model.
Why? Because you can't allow cheating, and you will always want some degree of prediction. Even if you didn't care about either of these you can simply choose to not opt-into those limitations.
But you get too many armchair network experts who have never opened a socket on HN saying that even things like serializing game data is too nuanced. Yeah, OK.
I'm sure there are alternative models that do exist, but it's been less than 25 years since that design took hold, and most significant real-time video games have used something exactly like it or similar.
Even RuneScape had serialized game event RPC that was reverse engineered over many years. Every game has something like that. You will always need to at least serialize something.
There's no point wasting time getting excited about your brilliant new product if I can't quickly verify that it will fit into the tech stack I'm operating in.
What's not explained is how a real-time game would handle prediction and latency. On the local network the Among Us clone is probably alright, but could have both input lag and unusual movement with any latency. From what I can tell there's an onTick that is moving the players on the server, but no local prediction or synchronization that usually makes these run well. Maybe I'm overthinking things -- most TVs have 50ms~ delays on standard settings and we adapt. This could have less.
Show HN: Hathora – Multiplayer Game Development Made Easy - https://news.ycombinator.com/item?id=30442072 - Feb 2022 (35 comments)
Seems rather orthogonal to what Hathora is.