Deterministic Lockstep for Networked Games

Deterministic lockstep is primarily used in RTS games, where the only data that is transmitted per tick are tiny commands like "player 1 is constructing a war factory at this location" or "player 1 told unit 6 to move to this tile." If we're ridiculously generous and say that a kilobyte of commands are sent per tick, then 120x bandwidth would be 120 kb/tick. 10 ticks/s is a common RTS tickrate, so that's about a megabyte per second. In practice, far, far less bandwidth would be used than this, given that this architecture scales down to 90's dialup connections (https://www.codeofhonor.com/blog/the-making-of-warcraft-part...), but even a megabyte/s is not a completely ridiculous amount of upload bandwidth for much of the first world (but sadly not the USA).

> Audio/video streams are a completely different problem - lost data is just lost. This article is discussing a very specific problem where no data loss can be tolerated.

I know that, which is why I separated deterministic lockstep from other solutions in my comment. You're missing my point, which is that soft-realtime problems aren't suited for TCP, and cannot be addressed by any single protocol because constraints differ.

The point is though that sending 15x more data is not an issue when you're only sending a few bytes per tick.

When optimizing, you always profile and optimize the bottleneck first. For a lot of people in a lot of situations, latency, not bandwidth, is the bottleneck, and that's exactly what this solution solves. Compared to e.g. Standard web usage, most games are actually very small amounts of data sent, but very high time criticality, and hence very sensitive to latency.

In the case of wifi and such this seems like a sensible approach, too bad its being applied end to end.

Not in this case - packets can be dropped by the route and the game player doesn't care as long as his connection is good enough to play. When it gets really bad the player will likely quit and the packets will stop.

If it is really desired you could implement throttling according to packet loss, but not in the way that TCP does it - by buffering and waiting - instead you'd just send packets every N frames. You can't do that if you're just using TCP since you don't know when packets are dropped.

I have good news for you. We game designers design these protocols to have a (low) maximum rate and a (low) maximum fixed length, because our games are even more performance dependent than the internet and we frequently have very poor or limited networks.

For example, one of Glenn's earlier protocols kept a maximum of 32 previous message IDs, and operated at a rate of less than 30 UDP packets per second. Unacknowledged mandatory messages were given up on after 1 second of loss.

And lastly: the amount of extra bandwidth that Glenn proposes consuming in order to provide quasi-reliability on top of UDP is about as big as the TCP header. So you could consider his protocol a degenerate case of TCP that just usually doesn't send headers.

This is not always true. Some amount of packet loss exists over WiFi and 3G and in this case this packet loss is not indicative of congestion.

I don't want to play a multiplayer game with hitches every few seconds. You can't fix this with "smoothing" with deterministic lockstep model but you could fix it by increasing the playout delay buffer from 100ms to 250ms, but that's adding another 150ms of latency to the player experience which is less than ideal. It's a much better option to use UDP in this case to get the best experience for most players.

Satellite or radio based ISPs, mobile networking, and if we get exotic, interplanetary communications all come to mind as environments which fit all three criteria.

You're not sending more packets, but slightly larger packets - as the article says, even 120 frames of data takes up 90 bytes. When, even with extreme repetition, you're sending single digit kilobytes per second over the Internet, it's unlikely that your stream itself is substantially contributing to congestion, unless you have some completely broken/saturated link that nobody wants to play on anyway. (One might want to send enough bytes per frame that such a buffer would start to bump up against packet size limits, but 120 frames is ridiculous. I'm not even sure where he got it from, since 60 * 5 seconds timeout = 300, not 120 - and you don't really want to time out after 5 seconds, since transient total outages happen when accidentally going out of range of a Wi-Fi network or whatever. Layer the opportunistic resending on top of a TCP-like slow resend mechanism for the worst case, and then there's no point sending each packet more than a handful of times.)

I implemented opportunistic resending last year for netplay in a branch of the Dolphin emulator. The target audience was a group of people playing a fighting game (Super Smash Bros.) at a semi-high level, so it was essential to keep latency to an absolute minimum. Even a few frames of lag is enough to substantially impede high level play (might be mitigated by prediction/rewinding if I ever implement that, but since fighting games are based on very quickly responding to others' actions, there's only so good it can get); the 100ms from the article, for example, would be totally unacceptable for anything serious. Even if it were feasible to implement client-side smoothing in an emulator, it would still be much superior to actually get the real data in time.

The question is then whether packet loss is common/significant enough in practice that opportunistic resending's effects are visible in any significant way. For Dolphin, the main advantage of using UDP over TCP is actually NAT traversal, so there is no reason not to do it, but it would be nice to know whether it helped. Unfortunately, I don't have data here: I didn't include code to explicitly record packet loss statistics or anything, and I haven't had the time to actually clean up the code enough to promote it to master, so there aren't many people using it anymore. Anecdotally I implemented opportunistic resending, there was feedback that it was a noticeable improvement; there would certainly be some confirmation bias in there, but the feedback was positive enough to convince me it was likely a real effect. They didn't necessarily have to be losing a lot of packets; as the article mentions, with TCP a single packet lost makes you wait something like 2*RTT, which is certainly noticeable if your input buffer is set to the lowest possible value that produces smooth play under normal conditions.