MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve
prdeving.wordpress.com
prdeving.wordpress.com
There's concerns around race conditions as you pointed out, message passing from client to server, and server to server, client hand-off between sharded servers. Those synchronization problems will haunt your dreams.
I think the biggest issue I still struggle with is tracking those ephemeral problems that only happen on one shard, or only when going between this shard and this shard, but not the other way. One useful trick is obviously message prioritization and different messages heading to different servers - though these days I'd put a message router in front of the shards and the router handles persistent connections other than the usual technique of direct connection I've employed in the past.
Contributed to an MMO game that involves waving light sabers around, another where you defeat the ultimate prime evil (though I was more on the fraud detection on that one), an unpublished MMO that unceremoniously died during the 2008 financial crash, a "shared world" game that involved animals, an open-world game that involves driving cars and running pedestrians over, a few "internet scale" websites, and am currently lead back-end on another MMO - though our database requirements are relatively simple this time around, but it is still again, read-at-start-up, write-only-when-necessary.
Carmageddon is not an MMO. :-)
For example, it took 5+ minutes to load for years, until some random guy fixed it for them [1]. Though I suppose that had little to do with the overall architecture.
Would you choose to author an MMO backend in one of those friendlier ecosystems?
Do you think there's value in having access to other Java applications, to embed as libraries of your grander "in memory" ideas?
You’ll frequently find Java, C#, Go, and Python as backend rpc/tcp game servers.
There's definitely some value in using more productive languages for the backend services, as they're not as latency sensitive as rendering.
Second Life / Open Simulator makes a big distinction between assets, inventories and area state. Assets (meshes, textures, animations, sounds) are immutable, and are stored more or less permanently. (There's a garbage collection batch job that runs monthly or so) Those are basically files. There's a vanilla web service running on an AWS web server, and Akamai, both heavily cached.
Inventories are like file directories. They have asset UUIDs and some metadata (name, etc.) Those are in a database, but that data is dynamic and not cached. Each user has an inventory, of course, and it can be huge. 50,000 items are not unheard of. This is a metaverse; you can build stuff.
Area state is in server memory for each region. That's saved periodically, once a minute or so. This is a backup file, not a database. If you wanted a more continuous save process, you could keep a log of recent changes on a different machine than the server. After a crash, reload the server state and rerun the recent changes. Area state is under a gigabyte per region (A region is 256x256 meters).
With a three level system like this, none of the levels are severely overloaded. The greatest data volume is from the asset store, and because that's immutable, it can be and is cached extensively. There are three levels of caches - asset server, CDN, and client. It's still a problem getting assets out to the clients fast enough, but with prioritization and concurrency, that's solveable. The inventory database is mostly-read, so the usual scaling techniques for mostly-read databases work. Area state is in memory. The main trick is taking a clean backup without visibly freezing the system.
Rewind 15-20 years ago and all the folks who wanted to make a game, their first game, and they want to build an mmo. None of them succeeded. Not 1. The only ones since were from people who knew the ask. Or had a crowdfunded ponzi scheme.
I guess it depends on how you define success, but I would posit FOnline [1][2] as a success story. FOnline is a fan made MMO of Fallout by a single guy, using the assets of the original Fallout 1 & 2 single-player games. Having these assets and also general game mechanics already finished definitely played a huge role in getting it to a playable state in reasonable time. Still, FOnline is a from scratch code base not a mod of the originals. Also it changed plenty of mechanics too, most notably being real-time while the original games were turn-based.
It's still being pushed forward even today after 20 years of development by this one guy but it was playable in late 2000s already. Peak concurrent players that I remember seeing was a few thousand. Definitely not AAA level, but way past simple multiplayer. Would have gone higher due to the hype at the time, but the server started to really struggle at that point. After a few years of being a closed source free game it got converted into a SDK and spawned a dozen new fan games using that engine.
Perhaps even more importantly, it was extremely fun in the early days. PvP gained you experience and all the other player's loot. Later on the PvP was limited due to PvE lobbyists, but perhaps it made the game more fun for PvE lovers.
Here's a random screenshot from my personal archive that shows a bunch of players on the screen at once. [3]
In any case, I view it as a great example of a single person MMO success.
--
I’m talking about 10,000+ players. Where you need clusters of servers and synchronization techniques.
There have been some small multiplayer games that have tried to pass as mmo’s but without the player base in the 10k+ range, you never encounter certain classes of engineering problems.
Your "synchronization techniques" pale in the face of destructible terrain.
Going by my logic, there are MOG’s and there are MMOs. Guild Wars 2, World of Warcraft, EverQuest 2, games where thousands of people can be on screen at once, where that server instance is synchronized with the rest of the cluster, for one seamless virtual experience. I’d say half of the self-proclaimed mmos ever really reach massively-multiplayer status. Games like Dota 2 and CS2 are played by millions but I wouldn’t say it’s an MMO because each match it’s 5v5.
Realm vs Realm or World vs World mechanics explain my perspective perfectly.
The original release of World of Warcraft had a limit of a few thousand per server too. It's only much later that this got increased. In the classic MMO era there's really only EVE Online that pushed to 10k and beyond, and never in the same star system. Single star system was limited to around 500 people for the game to still be playable. It's only later that they added time dilation which allowed for thousands to be in a single system at once.
This is an odd statement given that Ultima Online can out in 1997 and WoW came out in 2004. Are you referring to hobbyists?
If you are arguing they made it more than 20 years ago, so it was easier, I'd like to learn more about how things degraded in the early 2000s. I assure, I am not confused by words or math. :)
I thought most MMOs kept everything in memory and only offloaded to the database periodically. I clearly remember rollbacks to fix times (XX:00) when things went down.
Edit: Sorry, should have read the whole article before commenting.
There's so many of them. I have to assume they're spatially limited. You enter a zone, or an area, and the system loads up all of the quests located in that space, then it runs through to determine whether you qualify for them.
As for quests that you're on, that's a bit more straightforward, since you're so limited to how many you can carry around with you at any one time. Then, every event can practically just be brute forced across your pending quests to see which ones get progressed, etc.
But it was always a curiosity to me considering the magnitude of the quests available how most anything can trigger quest progress.
There's also the whole achievement system, which perhaps is similar in design.
In short, from what I know, yes, quests are stored in "quest log" fields of character data in server DB, and they are tracked by the clients and checked by the server. Some simple auto quests like "find this item" are not even tracked by server and only stored on completion. Since both client and server have all game data, the client knows about all possible quests and only shows to the player what is appropriate at a current state.
This is not necessarily that difficult, at least the first part. A lot of games will have quests be given out by a 'quest giver' character of some sort, or they will activate at specific interaction points on the map. You can do some cheap 'has-player-finished-quest' type of checks to determine if for example the quest giver has some sort of UI to indicate they have a quest available that activate when the quest giver first comes into view range. Quests with more initial conditions can hide their checks behind the interaction with the quest giver.
Doing quest progression can be a bit more challenging. You need to determine when to do the checks for progress, and also how comprehensive you want them to be. The more complex the check, the less often you can run it without affecting game performance. I've seen designers use all sorts of tricks depending on the specific quest. Interaction volumes that run checks, periodic ticks, on entity flash messages..etc.
> Then, every event can practically just be brute forced across your pending quests to see which ones get progressed
This only works for games that have a small number of active quests and not a lot of events. And with MMO's, you really need to be considerate of the accidental quadratic performance problem.
Trinitycore emulator can handle 10k+ players on a single server.
Thanks!
Tangent: Imho, the only reason it is good is because it's not as grindy and / or the community just didn't put as much emphasis on min-maxing things. GearScore was a thing of course, but theory crafting wasn't anywhere close to what we have now.
Played through pandaland, skipped warlords, came back for some of legion and then finally broke the habit.
Also how do they place all the NPCs and hook up the quests and lot tables? Do they scrape wowhead?
Yes they scrape as much as they can, but the originals of these (almost as old as WoW itself) were absolutely not 1:1 behaviour. By hardcore players noticing the minute differences, taking the time to detailed bug report, then the admins/devs looking with care, they've come pretty close.
Never underestimate a group of nerds with a passion.
the size of 'big engagements' that are still playable has gone from a few hundred to a few thousand in this timeframe. thou players tend to hit the ceilings and "playable" is a matter of opinion.
https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2017/Pres...
DAoC was basically a Linux box with a bunch of processes connected to MySQL.
https://www.gamedeveloper.com/disciplines/postmortem-mythic-...
> One the most impressive features of the Turbine engine is the continuous outdoor environment. This is made possible thanks to dynamic load balancing, which is a scalable serverside architecture. The easiest way to appreciate the need for dynamic load balancing is to consider the following scenario.
> Dynamic load balancing solves this overloaded server problem. Instead of assigning a static geographic area to each server, the individual servers can divide up the game world based on the relative processor load of each server. In the previous example, instead of remaining idle, all four servers would divide the load equally among themselves, ensuring the most efficient use of the hardware’s processing capacity.
Improbable tried that, dividing the world into regions but moving the region boundaries around based on player density. This worked, but apparently required huge amounts of inter-server traffic. The system was too expensive to operate. (Running it on Google Cloud with metering for every client/server transaction didn't help.) Five indy free to play games, some of them good (look up Worlds Adrift), went bust because of server cost.
Improbable then pivoted to simulators for the UK military, a much less cost-sensitive market. That worked, but they had way too much company and funding for that niche. Then they tried to pivot to crypto metaverses, two years too late, and hooked up with the Yuga Labs (Bored Ape, Otherside) crowd. Lately, they're trying to do something with US Major League Baseball. Their solution to the cost problem is to only run special events that last a few hours, for which they can short term rent some huge number of servers from AWS or somebody.
There's still no good off the shelf solution for this kind of scaling, with big worlds and big moving crowds. Epic and Roblox were making noises about working on this problem a year ago, but not much has been heard recently. Now both are in money-losing and layoff mode.
[1] https://www.reddit.com/r/wow/comments/9huows/ama_former_wow_...
[2] https://www.reddit.com/r/classicwow/comments/9fb2bo/john_sta...
WoW is a 2nd or if you count Meridian 59 as gen1, 3rd generation MMORPG.
It was not the 1st that had seemless maps, but IMHO it did that best. EQ2, while not having seamless maps, would be in the same league as WoW.
As Eumenes notes, the Asheron’s Call engine was significantly ahead of its time with the seamless zoning. The cost was high, though — we needed quite a few servers to run a world, I think many more than our competitors. There are business reasons why we weren’t quite as cost-conscious as we perhaps should have been.
The other factor involved in determining how often you persist is item duplication. If it’s possible to transfer an item between players without persisting state, and if there are known exploits that crash servers (not world, but individual servers), you wind up with an exploit that can duplicate items. But I’m sure that’s just hypothetical.
Then it will be too late because you will essentially have to rewrite half of your project while your userbase is leaving due to unplayable game. Better to make good architectural decisions from the start, and make small optimizations when needed.
I've also been working on an engine for the past few years if you want some code examples: https://github.com/Net5F/AmalgamEngine
Also important for most online games: any core gameplay relevant actions need to be server authenticated - usually you do RPCs from client to server, resolve the result, then broadcast it to the relevant clients.
Once you start caring about the performance the second thing you do is stick a cache layer of one sort or another in front of the database; the first thing should be making sure you have the correct indexes.
In any case, it sounds like a distributed cache problem. I wonder if you could just abuse redis for your game backend?
When I worked on a MMO with a five figure concurrent player count we got by fine with a single database server.
The much bigger I/O bottleneck were with the load balancers.