Hacker News Reader

Comment by xwdv | Hacker News Reader

How do MMOs maintain and update state for tens of thousands of players and then push those updates from the server back to clients efficiently?

View on HN

As said before, different type of MMO use different technologies. I used to work on the server part of a (yet to be released) MMO game; we described the technology a bit in the following video [1].

A few things to consider:

- the amount of information to be sent from the server grows quadratically with the number of clients (eg. if 100 players all 'see' each others, you naively would have to send 100*100 updates per server tick); the data received by each client only grows linearly, though

- obviously these are kept in memory, and persistent data is flushed if needed to a permanent storage

- one can reduce the quantity of information by:

+ distance: when a player (an object) is far away, it may get invisible (too far) or degraded (updated only every N ticks, low quality version of the assets sent/displayed)

+ number of entities around: when a lot of players are grouped closely (think a crowd), you can also degrade the information for those further away, even if they are close to the viewer in absolute value, since they are 'hidden' by others.

+ compression tricks: send deltas instead of absolute values for position (smaller numbers can be compressed more than larger ones), don't include the properties that did not change, etc.

- at some point, you (may) need to split the work on several servers, and have an efficient algorithm to split the game space, and synchronise your servers

[1] https://www.youtube.com/watch?v=QeZtqoydXpc

Depends massively on the type of game. In general the two techniques are heavily compressing the data format sent over the wire (e.g. using 16 bit floats instead of 32 bit where it doesn't matter, sending 3 element quaternions instead of 4 and recalculating w), and only sending what the player needs to know based on a radius or some other criteria.

Eve Online slows the entire simulation down when huge battles are happening, but they also split simulations across servers based on which system you are in.

Modern WoW shards plays heavily within zones, which is why you will almost never run into the same people while adventuring. They also attempt to limit how many players will be in the same shard at the same time. However there are ways around that through grouping, so when larger battles start happening things get laggy. You press buttons, they are queued up, and can take upwards of a second to show their effect in-game (they must go to the server and back).

Will this change with technology like Google Stadia? Could a massive compute farm calculate all of the state for all players, and then just stream video data down?

My guess is no - you'd want the rendering to happen on the edge close to the player, but with a sufficiently advanced compute topology maybe some tasks can be farmed down the graph?

My understanding is that the client-side isn't generally the bottleneck for MMOs, it's all the server-side computation that has to happen to make it all run. Calculating behaviors, physics, etc for hundreds or thousands of players simultaneously is really hard, and Google Stadia doesn't change that in any meaningful way.

Edit: Well, now that I think about it, it depends entirely on the game; PlanetSide 2 is an example I can think of off the top of my head that brought my computer to its knees during big battles. But most MMOs aren't exactly pushing the envelope of graphics quality or physics simulation, so it's not as big of an issue.

Sharding, and many other finer (game and architectural) design and implementation details. Few MMOs actually have millions of simultaneous players in the same instance of the world or boss or raid at the same time.

The MMO-lite I worked on was made to be as horizontally scalable as possible, and had extreme incentives built into the gameplay to push players towards actions and areas of the game universe that would more fully leverage benefits of the scaling trade-offs we made.

Being able to segment and slice the outbound data from your clusters (and/or what the client requests) - that is, the data and game state equivalent of occlusion culling - goes a long way.

In memory. MMO servers store all world state in memory, and each server has a capacity usually in the 10,000 player range.

You'd be impressed how fast computers actually are -- 10,000 objects isn't actually that many. Since the server knows where all clients know, the server knows what the client can see, and the server only has to update the client with relevant information, like the 100 closest players to it.

Generally the server does not in fact know what all the players are doing. They're sharded at the most granular possible level (typically a zone or instance) so any server instance knows about no more than 50-100 players.

> 10,000 objects isn't actually that many

Yeah but these 10k objects are containing themselves many more objects (item inventory, proximity mobs etc).

Not just that, but you have to do physics and other calculations on each object and even between objects many times per second. So if your server handles 20 ticks per second, you only have a budget of 50ms per tick. Even on a beefy, many-core server, that's a lot to do in not a lot of time, and you can't afford to have any inconsistencies with performance (like GC stalling for several ticks).

Clients in MMOs don't need the entire state, just the state in a small area around the player's location. Some times it works like chat channel with a limited number of maximum players. When the limit is reached, player are put into different channels, so the state of one channel doesn't need to communicated to other players.

Same way you don't render the whole world each frame: https://en.wikipedia.org/wiki/Potentially_visible_set

Also 3D data structures, which http://realtimecollisiondetection.net/ is still my go-to book.

Usually, you keep all the state in memory and save at specific points/actions to a database. Then, not all state is shared with all thousand players, but only within a visible range. And then, not every kind of state is shared with other players. In the end it’s manageable with simple TCP or custom UDP connections.

Not a game dev but I guess you could go quite far with just an IRC server. Crosshatch your world in chat rooms, with a bot in each one.

A player sends a command to his current chat room; the bot validates the player position and current action.

Other players listen to bot responses in all nine chatrooms around them.

Payload trimming is always the answer. Only update state that has changed...

You can do a lot with the right, simple architecture and a programming language at the right level of abstraction.

I bet you could answer 99% of Stackoverflow questions with this generic answer ;-)

Indeed ;-) But I bet 99% of those wouldn't need to be asked if people hadn't overcomplicated things in the first place...

Multiplexed sockets. Dead reckoning.

I remember a lot of dead reckoning happening in the old Runescape, once upon a time. Many moments of your player running in a direction then getting rubber-banded back because they got interrupted or something.

Dead reckoning is a bit harder when your game isn't using a perspective based camera (2D or orthographic). Otherwise you can use visual tricks to hide the rubber banding from a distance. Though when in 2D the solution is to interpolate between waypoints.

What actually happens in Runescape is that certain animations override the running animation. When the animation then ends, the game compensates by making your character move much faster than usual, which can look like zipping all over the place. This is all client side of course; server side your character moves as normal.

Extreme example of someone using this in PVP: https://www.youtube.com/watch?v=kIFWSMW6Lzo

Reply on news.ycombinator.com