John Carmack on QuakeWorld latency and business model (1996)
raw.githubusercontent.com
raw.githubusercontent.com
If you weren't a gamer (or alive) at the time when quakeworld came around, you might not appreciate how amazing it was for multiplayer games on the internet. On dial-up, you were lucky to have 150ms latency. Before client-side-prediction, that latency applied to every action you took in game, including player movements. Hit the up-arrow, and you wait 150-300ms before the game responds and moves your character forward. CSP really was an amazing break through, and made multiplayer action games feasible on the internet.
This is particularly relevant now, that we are entering the era of cloud-based streaming game platforms, like Stadia. The latency problems of the pre-CSP 90's will be rearing their heads again. Its going to be interesting to see how these same problems will be tackled in this new context. Internet speeds are higher now, but so are our expectations.
I doubt we'll have the nice, simple dotplan files that Carkmack left for us to read and remember, from all the SRE's at Google, sadly.
I'm not sure if it's funny or sad that there's more key press latency typing into most local Electron apps than connecting to a Quake 3 server 200 miles away back when I had 56k dial-up in 2000.
If you want to fast forward to today's internet, with an average internet connection it takes around 150ms to ping a server in the Netherlands from California. That's over 5,000 miles (8,000 kilometers). Somehow a local key press has the same latency with certain code editors. What have we gotten ourselves into.
"The mess we're in" famous talk by joe amstrong should be transformed into a website listing all of those absurdities, as a way to public shame the culprits.
For anyone else: https://www.youtube.com/watch?v=lKXe3HUG2l4
Fantastic watch, thanks for recommending. Sad to hear that Joe Armstrong passed away a few weeks ago.
// This is Makepad, a work-in-progress livecoding IDE for 2D Design. // This application is nearly 100% Wasm running on webGL.
If you want a fast editor, switch to Sublime 3.
its more than necessary but not as bad as you are implying
Theoretically the server could speculatively render and transmit a number of different "potential future frames", and the client throws the wrongly predicted frames away.
That's a nice way to burn even more energy and bandwidth though ;)
VR actually introduces further latency problems. With a TV, your typical PS4 can cover up latency issues with spectacle, as you mentioned. That's a big reason I suspect so many console games feel like a movie today, with tons of cutscenes and quick time events. I've been playing a lot of Bloodborne lately, and while it's an action game, it's still incredibly slow compared to something like Quake.
But with VR, when you move your head you expect the world to feel as if it is real. A TV is artificial, and latency is not an intrusion in the experience. But with VR, latency is felt on a deeper level. Potentially resulting in headaches and nausea.
The funny thing is that John Carmack is riding the state-of-the-art decades later on this front as well: https://www.wired.com/2013/02/john-carmacks-latency-mitigati...
True, but don't overestimate the importance of "serious gamers" to the industry's bottom line. IIRC, mobile gaming is now making more money than all PC and home systems combined.
Another issue is that machine learning/AI don't predict rare events, like earth quakes. So even with all the knowledge in the world, it won't predict a rare creative move of a player.
I say some but I do believe a large enough volume of data can improve the performance of this class of input/states.
Stadia could sell it as an add-on to players which not only don't want to play their games themselves, but also aren't satisfied by watching other people play through their games on YouTube or Twitch. With this add-on, they can finally watch themselves play through their games, without having to lift a finger to, you know, actually play!
Call it Sonic Mnemonic.
I'm also skeptical about the ability for a model to generate predictions without having too many mispredictions to make it viable.
And those types of game are the ones that are suffering most from input latency.
In a twitch shooter, you mouse over a visible opponent. Would the AI be more likely to pre-emptively pull the trigger for you? Or more likely delay and swing the aim passed the opponent before shooting at air? AKA Is the training set for predictive user actions based on experience of players better or worse at the game than you?
My take is that it'll be the a primary competitive point, nearly as important as the library available. Companies that can deliver the service without introducing these issues will succeed and ones that cannot will fail. If nobody can reliably crack it, cloud gaming won't take off.
Back in the 90s there was no viable competitor aside from LAN parties, and those weren't available to you every evening.
Also I agree that is not a given that "Cloud gaming" will take off. We have emerging VR were latency is absolutely critical, even more so than for FPS e-sports
source: German who never had a 56k modem but started with ISDN in 1998 and can't really remember a ping > 100 on EU servers ;)
Telekom used "interleave" by default, which provided very slightly faster download speeds. And caused about 70 ms latency to the first hop.
I had to contact them and ask them to change my ADSL to "fast path". This dropped the latency to maybe 20 ms (IIRC, my memory might fail me on this number).
I think ISDN was about 40 ms to first hop, but again, it's been a long time.
If you don't know what that is, Valve has some good documentation on it (relating to Source engine, but it's the same concept).[1]
Latency was still a problem even with CSP because we didn't yet have lag compensation on the server. So although you got an authoritative server (no cheating) and instant inputs (no round-trip wait), you could still shoot someone on the client and miss them on the server. You had to shoot ahead of them, further if your latency was worse. As far as I know, Source engine was the first to add lag compensation, which has the server rewind the other players to be where you would have seen them on the client.
Since client-side prediction can be performance intensive, and lag compensation can be difficult for anything beyond simple raycast checks, many games still don't do both. Team Fortress 2 for instance does lag compensation on basic guns but not on things like the soldier's rocket or the pyro's flamethrower. Some games simply allow the client to decide who they hit, which removes the need for either solution, but totally opens things up to cheaters without some very careful server-side validation.
[1] https://developer.valvesoftware.com/wiki/Latency_Compensatin...
There is still some documentation left on the internet: https://www.ra.is/unlagged/faq.html https://openarena.fandom.com/wiki/ModCompat/Unlagged
“GoldSrc is a game engine developed by Valve Corporation, first showcased in the 1998 first-person shooter game Half-Life. Elements of GoldSrc are based on a heavily modified version of id Software's Quake engine. “
This is one of the most influential paper for multiplayer games. ( probably #1 ). Tribe was the first game to have client side prediction and rollback, this is what most games use nowdays.
The source of the released game is available, so it should be possible to check, except that it's a huge mess (check out BUILD.C).[1]
Seems weird that no-one has actually verified this when it could be the first game ever doing CSP. The only other sites I can find mentioning it are just quoting your linked Wikipedia page.
Because of the unique challenges, they did a lot of synchronized client-side simulation (based on minimal state transfer over the net).
... and apparently encountered a lot of the issues with doing that (no random anything!!!).
Gives you really low bandwidth reqs (and randomness is still possible assuming you just seed things the same on all machines).
But also means you might have to wait 50-200ms (or some arbitrary sliding window depending on network conditions) until your click actually registers ingame as a move as commands are scheduled to be processed far enough in the future (some number of command turns in the future) that you at that point would have received every other players commands for that given turn and thus be able to execute all commands for all players locally for that turn, which is not ideal if you're playing a twitchy shooter, but alright if it's an RTS.
In fact, in some ways Build had even managed to scoop Quake, according to Ken: "People may point out that Quake's networking code was better due to its drop-in networking support, but it did not support client side prediction in the beginning," he explains. "That's something I had come up with first and first implemented in the January 1996 release of Duke 3D shareware. It kind of pisses me off that the Wikipedia article on 'client side prediction' gives credit to Quakeworld due to a lack of credible citations about Duke 3D."
domovethings()
fakedomovethingscorrect()
fakedomovethings()
getpackets()
Overview of Source engine networking: https://developer.valvesoftware.com/wiki/Source_Multiplayer_...
Ways they combat latency: https://developer.valvesoftware.com/wiki/Latency_Compensatin...
Lag compensation (server): https://developer.valvesoftware.com/wiki/Lag_compensation
Prediction (client): https://developer.valvesoftware.com/wiki/Prediction
IIRC QW's client side movement prediction didn't attempt to predict the effects of, for instance, shooting a rocket at your feet/rocket jumping, which meant on high latency connections there'd often be a bit of a delay before your view caught up with where the server thought you actually were. This also applies to rockets fired by other players. Lower latency had the effect of not only making it easier to fire rockets at enemy players, it also made everything seem smoother when people fired rockets at you.
Later versions of QW created from the GPL source release have introduced lag compensation for hitscan weapons, but I'm not aware of any attempts to do so for projectiles.
Projectiles like rockets apply a lot of forces to the player which are not easily predictable. If you fire a rocket and client-side prediction has it leaving your vicinity without collision but the server has someone stepping into the path of the rocket so it explodes close enough that the explosion exerts force that changes your position, you're going to have a really bad misprediction on the player's position, which of course can get waaaay worse if the player launches another rocket while the first rocket's effect on player position has not been sorted out properly.
The server has the authority on where you are, to prevent you cheating by sending it bogus positions. You send inputs (not your position) and the server gives you your new position back.
Problem is of course it takes time for the inputs to go to the server, and for your new position to come back. So in a naive implementation you press to move forward, and don't actually move for a while while you wait for a return signal.
So instead, say it's tick 100 on the client and you get a position from the server marked as tick 90. Instead of moving the player to the tick 90 position, which is in the past for you, you take that and "predict" ahead (a.k.a. run the simulation) 10 more ticks to get to tick 100. Including re-running any new inputs you did during those ticks.
There are situations where the client may predict differently to what the server actually does - maybe you bumped into another player and due to latency they were in a different place for you than for the server. In that case you'll either get suddenly teleported to a different place as the server data keeps coming through, or if you're lucky the game will kind of smoothly correct you over a few frames.
Yep
"He gave the example of increasing the refresh rate on the Gear VR during development. He was working, at that time, with its Galaxy S III phone. Android triple-buffers graphics, inducing a 48 millisecond delay into the system -- making VR impossible.
Carmack pulled apart Android to hack that out. Though he'd written several emails to Samsung in attempts to convince them to give back that buffer, "It's easy to argue against an email, but it's much harder to make the argument when you can have the two things and stick them on your face and, 'Tell me this isn't better.'"
> "Focused, hard work is the real key to success. Keep your eyes on the goal, and just keep taking the next step towards completing it. If you aren't sure which way to do something, do it both ways and see which works better."
He'll even do Samsung's work for them to fulfill the purpose of that quote.
Finally networked Quake was an amazing experience when it came out. I had started my first job in an ASIC design company. All the computers were Solaris based Sun workstations, yet we were able to get network quake up and running on them! I just can't imagine such a thing happening today but maybe that's more to do with me being 20 years older than anything else. For sure no one is porting game engines to Solaris these days ;)
I really like his ability to take out code and 'shoot it'. Back to the drawing board. It's a quality of a great engineer, to be able to reflect on what's been done and admit it's not good enough.
I'm not one for hero worship, but I think that's as close to engineering zen as one can get.
Aka 'I'm brilliant. I thought I was doing the smart thing. Turns out reality was otherwise. I'm changing my approach.'
Maybe Carmack will wake everyone up again and tell them to stop copying his ancient hack two decades later.
Second, it's also possible to end up through matchmaking lobbies on a server halfway around the world from one or more of the players in a given match. At some level the limitations of physics still result in noticeable latency and CSP is still mandatory for any fast-paced real-time online multiplayer game.
Players that are separated by thousands of miles should not be playing competitive FPS games together. It's physically impossible to make it a consistently good experience.
This would be debilitating in an online game without CSP where even the latency introduced by screen buffering and vertical sync may make or break yor game, and in reality the latency is of course much higher, usually around 100ms or higher. Check your assumptions: https://wondernetwork.com/pings
the bottom line is that I was working with the wrong basic assumptions for doing a good internet game. My original design was targeted at <200ms connection latencies. People that have a digital connection to the internet through a good provider get a pretty good game experience. Unfortunately, 99% of the world gets on with a slip or ppp connection over a modem, often through a crappy overcrowded ISP. This gives 300+ ms latencies, minimum.
So he designed for 200ms and then the real world was 300+ so it didn't work. Your 100ms would then work fine in that context. Maybe games have gotten more complex or our expectations higher so prediction is still worth it even on <50ms connections. But OPs point that the assumptions underlying Carmack's redesign have once again shifted seems correct, at least for that specific design.
By my numbers (though as you can clearly see from the link I posted, >200ms latencies are not uncommon), and Carmack's assumptions in 1996 about what latencies would be acceptable.
> So he designed for 200ms and then the real world was 300+ so it didn't work. Your 100ms would then work fine in that context.
He designed for <200ms which by the frame of reference he cites (T1 connection) probably was a lot lower than 200ms on average.
> Maybe games have gotten more complex or our expectations higher so prediction is still worth it even on <50ms connections.
Quake was not a very complex world, but more important (to latency) is that it's a fast paced game by today's standards (IME). It seems more likely to me that perception on what an acceptable hand to eye latency is has changed, just like the perception on acceptable frame rates. To me, joining an online shooter with >100ms latency (even with prevalent client side prediction) feels debilitating, especially if other players have lower latencies.
> But OPs point that the assumptions underlying Carmack's redesign have once again shifted seems correct, at least for that specific design.
Yes, I agree that PPP and SLIP are not common any more. High latency unfortunately is, and client side prediction is still an effective mitigation strategy that makes a huge difference to online play. The specific design used in Quake has of course been superseded.
Those numbers match my experience a long time ago with Quake3 based games where you'd pick servers that are local to you and get 10-40ms latencies.
> He designed for <200ms which by the frame of reference he cites (T1 connection) probably was a lot lower than 200ms on average.
I didn't forget the < sign. Designing for <200ms means you can handle the 200ms worst case. But maybe he meant 200ms worst case but much lower average like you are suggesting.
> It seems more likely to me that perception on what an acceptable hand to eye latency is has changed, just like the perception on acceptable frame rates.
Right, makes sense that the expectation today is much higher.
>High latency unfortunately is, and client side prediction is still an effective mitigation strategy that makes a huge difference to online play.
Just for curiosity, how low does it need to go for prediction to no longer be useful? I guess if you want to be able to have players from all over the world play each other then you have to deal with 300+ ms of latency. But if you are willing to do servers on each geography it seems 10-30 ms would be feasible. Would that be enough?
10-30ms is a bit optimistic, depending on how large you define a geography. Many ADSL/VDSL connections, best case, start with 5ms of latency, and often higher (say 20ms) due to interleaving. Cable tends to be around 10ms IIRC but can suffer from significant jitter which makes things worse. So for a lot of players the servers would need to be in the same city to achieve that target.
Just checked my fiber connection and it has 2ms of latency. google.com is 18ms away. I'm in Portugal and seem to be routed to a Google server 400+ km away in Spain. So my memory of playing on 10-20ms to local servers in Portugal is probably accurate. Covering the US or the EU with enough servers so everyone is never more than 30ms away seems feasible. Within easy reach of a startup with just ~50 VMs across different cloud provider locations and for popular games easy for users to setup regional servers themselves.
Packet loss is extremely low, which is easy to tell using a builtin netgraph.
Screen buffering should be reduced as much as possible and vsync is not something competitive players would ever want. Pros all disabled it in Quake Live for sure.
Players on modern connections that are playing on servers in their city do not need CSP.
In practice it's not that bad to have CSP if it's not really doing much (due to all players being low latency) but I know from playing NetQuake vs Quake Live with low latency that it's nicer not to have it at all.
CSP is still relevant as far as I can tell. Yes, connections in a lot of countries are now on average lower latency, and yes on average CSP might not be required.
However, it's not uncommon to have a couple of high latency connections on a game server (player in another country, roommate is running bit torrent, or just a poor connection). You can't always guarantee low latency, and sometimes connections are interrupted temporarily and packets get lost (assuming UDP).
Short of full sync more often, CSP helps make these unpredictabilities a little more tolerable and transparent for the player.
Your average multiplayer today is significantly more complex than Quake. More data needs to pass through, at higher rates. The network is still very much a bottleneck.
A lot of the extra bandwidth usage is also probably in large part due to sloppiness. John Carmack did some impressively smart things in Quake Live to reduce bandwidth usage. I doubt most modern games like PUBG are anywhere near as optimized.
Then take out CSP from their games and watch what happens.
CSP will always be necessary for any Internet-based game. Only LANs have low enough latency to not need it.
Which is exactly what competitive Quake Live players do and I suspect CS players probably do. Every player has 15-40ms ping and it's amazing.
Your requirement that games ping-restrict servers is an absolute non starter: unless the game is already guaranteed to have hundreds of players active and available near any given player at any given time of day, and evenly distributed redundant always-up servers that scale for demand near every player cluster across the planet. I think CSP is a far more realistic solution to giving a good player experience than magicking player base activity and distribution
What you get in exchange is a stable and predictable competitive game. I, and many others, experienced endless frustration in PUBG due to players warping around corners and shots that were clear hits missing due to CSP. And I'm not one to complain about this kind of thing unnecessarily. It really sucked.
The PUBG people didn't even geolock their servers at all last time I played, so you could be playing with people with 300ms+ ping which is just miserable.
For less successful games maybe you have to increase the maximum ping from 40-60ms to 80-100ms or something but it should be kept as low as possible. The more players per city, the lower you can make the maximum ping.
I realize that this might make it more difficult for people in Australia (or whereever) to find a game but it seems even more wrong to make the experience bad for everyone all the time. And at least when they do find a game it will be a great experience instead of terrible.
Of course, there is an upper limit on how much latency CSP and lag compensation can compensate for, and this varies by game. One game might work well up to 100ms, while another might work well up to 50ms.
> I realize that this might make it more difficult for people in Australia (or whereever) to find a game but it seems even more wrong to make the experience bad for everyone all the time. And at least when they do find a game it will be a great experience instead of terrible.
Here you're failing to account for an even more important factor: ability. Locking matches to such low relative pings would greatly restrict the number of players who were eligible to play together. That would greatly increase the skill gap between players, leading to a much worse experience.
Consider, e.g. Overwatch, which has all the Overwatch League pros on the west coast, leading to a skill gap between top players on the east and west coasts (which are usually matchmade within their own region). The more you compartmentalize players, the less likely that good players are going to be able to play with other good players, and vice versa, leading to a frustrating experience for everyone. (Not that Overwatch has good matchmaking in general. It's pretty bad, IME.)
Issues like getting hit after getting behind cover is frustrating, but it's going to happen sometimes--it's the nature of online games. But nothing is more frustrating than having teammates who aren't on the same level, or who don't have the same goals. It really makes the game feel like a complete waste of time. Matchmaking is definitely a harder and more serious problem than network latency now.
Probably more like 600, even semi-popular games using server locking (like Blizzard's Heroes of the Storm) struggle to complete matchmaking in under 10 minutes for 10 players. Matchmaking is a complex and challenging problem. Constraining MM to both servers <80ms away and clients that are all within 80ms of said servers, barring players with slower connections from playing the game, is a great way to ensure your game will never have PUBG/Fortnite levels of popularity.
As a game developer I'd rather people be able to play the game at all and feel playable than they all have only The Optimal Competitive Experience of Getting Headshots 100% of the time, after 10 minute queue times, or nothing at all. I would also like people who live in places where they can afford a computer, an internet connection, and my game, to play my game, and not just the continental US and EU metropolitan areas. Vietnam, Thailand, China, India are huge markets for games even with questionable internet quality.
Thankfully, most game developers and I concur on this so we all have more games to play. Maybe a compromise would be to allow players to check an option to only match with high connection quality servers/players when queuing, but so few players would check this option that one who does will have a relatively astronomical queue time to the majority who won't.
edit: for the record, I believe most such games running an actual tournament will use ping restrictions (or just run a LAN). Many also allow dedicated hosts to post servers with their own restrictions and constraints on who can join the server. I think that's a fine compromise. CSP shouldn't harm the experience if the latency of all the players is low enough that the prediction is almost instantly overwritten with server-side truth.
Then followed by Carmack's 90's internet experience: "Ok, I made a bad call. I have a T1 to my house, so I just wasn't familliar with PPP life. I'm adressing it now."
:p
With NetQuake you could learn to predict everything in your head over time, which is the primary skill competitive players use. No one has a super fast reaction time (even competitive gamers are ~200ms), the best players simply know what's going to happen next better than other players.
I did play thousands of hours of QuakeWorld (Team Fortress mostly) but it was never as fun or competitive feeling as NetQuake.
And once I got an ISDN connection it was even worse to use QuakeWorld because with a perfectly reliable (low jitter) ~50ms ping on NetQuake everything was incredibly smooth. Watching players with 150ms+ ping ("HPBs") warp around the map while you tried to shoot them was no fun.
Today far too many games rely on bad client side prediction. The last game I played seriously was PUBG and they put all of their servers in Ohio. Being on the west coast with 80ms+ ping it was just a terrible experience all the time, but they apparently (and stupidly) assumed it would work well enough due to client side prediction. PUBG could have been so much more fun with ~20ms California servers and a < 60ms ping restriction, which is how most Quake Live servers operated when I used to play that game.
The way to make online competitive games as good as possible is to embrace low latency connections now that most people have them and focus on placing servers in as many cities as possible. The speed of light can't be overcome no matter what we do, so the solution is for people to play with other people that are within ~500 miles.
I hold out hope that eventually netcode authors will realize this fact and finally re-create the incredible long-lost solidity of NetQuake. All they really have to do is stop being so damn clever!
Classic! Reminds me of the time I told my dad I wanted to make websites for a living and he replied "hey, that's pretty cool, but I don't think there's enough work available to make it a full time career". (This was circa 1996)
It was great though. It meant you could bunny hop around corners at extremely high speeds without running into walls.
What an elegant client-as-terminal I have for this incredibly popular physics simulation! Oh no! it takes up too much bandwidth. Welp - time to throw out the old lock-step game-loop architecture, and allow each client to stream incremental updates
I'm sure many contemporary engineers had similar problems, and considered the same approach, but either shrank-from or gave-up on rustling together an elegant solution. Like many problems in the early days of 3d/internet gaming, Carmack seems to have rapidly and nicely solved it with imagination & experimentation, all informed by a broad and deep understanding of data structures, & algorithms, and presumably an industry-leading amount of experience.
https://www.quakeworld.nu/forum/topic/2356/qw-physics-vs-nq-...
I wonder if there'd be any interest in reviving this. I think it would be cool to have something like an RSS feed of the .plan files from various developers I respect/follow. These days you have to settle for reading their Twitter+GitHub issues.
(There were actually two such files; .project was used to give a high-level summary of what you were working on, while .plan was used to talk in detail about what you were currently doing at the moment.)
https://www.bluesnews.com/cgi-bin/summary.pl
Click on a company, a person and then use the drop down to view older entries.
In some ways they reflect the way I've gone about a days work more clearly than other project management/bug tracking software I'm required to use.
The original source of this material (and the other articles) was John Carmack's .plan file. See https://en.wikipedia.org/wiki/Finger_protocol
The .plan file is by its nature ephemeral, as the user can change it whenever. But the content was archived.
The GitHub is just the latest mirror.
> Will the next historian have to unearth it from github just to copy it to the next platform?
Maybe!
That does seem to be the case here. I had re-hosted these .plan files from a website – which has removed them – that had re-hosted them from Shacknews – which had also removed them.
GitHub seemed like a quirk-and-dirty way to re-host these while giving others an easy way of duplicating the original files so they’re less likely to be lost in the ether.
Sounds like that is what he did with Quake 3 (don't know about Quake 2). At least in Quake 3, the whole game logic is running in a so-called 'Quake Virtual Machine' file (.qvm) that is exactly the same on server and client (even cross-platform compatible). That way he doesn't just predict the client movements but the whole game world.
The Swedish qwctf scene at least was very clearly divided in 20ms people who went to or worked at, a university, and 150ms people who played from home.
There were some 50-60ms ISDN people too but I can only remember a few.
The next job I had was in a much bigger company, and the office I was in had equivalent of 70+ T1s. Most of those were to support the phone system though.
If the stars aligned and Carmack decided to get back into making new software-based gaming experiences (especially around the DTX/ATX area), I would be incredibly tempted to investigate opportunities.
> Its definately cool, but it is uncertain if people can actually make money at it.
I agree with his use of a double etc there.
[0] https://www.reaktor.com/blog/why-csp-matters-i-keeping-thing...
https://web.archive.org/web/*/https://raw.githubusercontent....
It creates .warc files, which are supported by Internet Archive. I don't know if it integrates uploaded .warc files to Wayback Machine, but Archive Team uses this format to upload manually archived sites to Internet Archive.
https://addons.mozilla.org/en-US/firefox/addon/save-page-we/