Postmortem of Service Outage at 3.4M Concurrent Users
epicgames.com
epicgames.com
There's a difference in the level of respect each company gives for its customers. I play PUBG a lot, but I want to see Epic win in the long run.
They don't do QA, and their 'deploy process' takes everything down for hours. It's embarrassing.
This reminded me of this classic: https://engineering.riotgames.com/news/automated-testing-lea...
Sounds like Riot have a good head on their shoulders about how to handle testing games.
If you come from a different domain, it's important to realize that testing games is much more difficult than testing a lot of other types of software. And much like other kinds of software, when it's done well it looks easy.
I mean, what did you expect?
After that, he moved on to making his own game with a supporting development company fully centered around his ideas at his direction with creative freedom that was not limited with being a mod (ARMA 2/3) or under the restraints of a larger company (SOE/Daybreak with H1Z1).
Brendan is largely the progenitor of the video game battle royale mode, and his name carries a lot of weight in the genre itself.
2) Investing in logging early, so that your system is giving you metrics as it grows
3) Understanding that logging and metrics are a conversation that happens with a system over time. You don't do it once and then are done, it's like teaching a child to talk. You have to leave room to expand, and be actively building a 'vocabulary' in your logging and metrics that makes your day-to-day simpler.
4) KISS -- at least on the "big stuff". Your high level architecture should be 'clean' in the sense that issues can be isolated and readily assigned to a subcomponent. I find that isolated communication points that let you bifurcate responsibility are key to quick deductions. Ie the backend publishes data to a store, front-end reads data from that store, so if the data store agrees/disagrees with either you know exactly where the problem is.
two very different games fundamentally
However, as for:
Why does it need to be a zero sum game with one winner
Your desire to see them suffer because god forbid the servers go down for five hours is an example of the shitty toxicity game developers have to deal with from fans nowadays. Blue hole isn’t Epic, they don’t have the resources or the comparable talent, but they made a really fun game. Isn’t that enough for $29.99?
A "battle royale" if you will.
When Epic started the UE4 project, they promised to take care of us gnu/linux users. I bought in the second I could. Immediately they started ignoring us. The first year and a half or so most of us Linux people were using a community fork because Epic refused to merge changes. We banded together and we're forced out of their irc and had to form another channel. I tried to be understanding, at first thinking they were low on resources. As time went on and games like Paragon (which I payed $20 for at release and which has now been abandoned, and I'm still waiting for my refund for over two weeks), and Epic started showing off how well things were going, they still basically abandoned us. There is still no marketplace/launcher on Linux, so to retrieve my $500 worth of assets I have to use a windows computer. Major bugs persist in all branches, and not just for the native editor... games like pubg would love to ship for Linux if the crosscompile tool gain wasn't an exercise in cryptic puzzle solving (which is why so many UE4 games are windows only not by choice. All pleas for more resources and love on the forums are met with comments about marketshare and how unworthy Linux is of their time and resources.
They promised us all this love and then after many of us spent lots of money on them they just ignored us. I'm thankful for the attempt for a native Linux editor, but crashes and uncompilable projects have essentially halted dev for me to the point I'm having to consider godot and blender even though they aren't nearly at feature parity, and due to epics licensing all that money on assets is wasted since those assets can only be used in UE4...
I love UE4 when it works. Blueprints, the animation and rigging system, the camera system are all wonderful to work with. I want to use it... but I'm feeling increasingly taken advantage of by Epic.
Tim, if you're reading this, how about a post mortem of the abandonware that is the Linux editor?
That's between $572,000 (500 instances, 30 days) - $2,863,800 (2500 instances, 30 days), per month at current prices, and seems like it's only for one aspect of their infrastructure.
That seems .... excessive? Is that a typical spend with a game server system like this? That does seem to suggest that once this becomes less than profitable, it's all going away ...
Those costs can easily be cut in half with reservations. Its likely there's a lower-bound of the number of reserved instances they use as a baseline performance guarantee, then they use on-demand to scale above that.
No one at their scale actually pays the list price.
Basic math: There are ~100 players in each game. At a 30 tickrate, that's 3000 RPS per game minimum. Each of those requests likely involves a number of 3D math calculations, including hit detection, collision detection, real-time cheat detection, etc. Those updates then need to batch back to the players at 30 tick. All of this needs to happen with as little eventual consistency as possible; a difference of milliseconds degrades the player's confidence that the server is correctly calculating what is happening in-game.
Point being; multiplayer game programming is an entirely different beast than normal web programming. The same rules don't apply. Its an n^2 problem where every additional user in one "lobby" actually increases resource utilization exponentially because you need to update every other user's game-state with the actions of that new player. Additionally, Battle Royale style games are the most demanding multiplayer games ever created. Games like WoW have way more players per realm, but servers only need to worry about the interactions of a select few in your surrounding area, and it doesn't have as stringent real-time requirements. Games like CoD only load in 5-20 players.
Blizzard is absolutely stringent about their real time requirements for WoW. It's crucial for PvP, Raids, etc.
Fortnite doesn't need to broadcast nitty player info for players on the other side of the map to you, either, I would think. That should definitely be optimized away
As for your last comment it's wrong, a game like Battlefield is probably more demanding compare to a battle royale type of game, a FPS like Battlefield runs at 60hz and is using a lot of physics everywhere at fast pace, battle royal games don't have that much action going on.
For example you could have dozens of late stage games that combined have less players than a single, just started, 100 person game.
I'm pretty sure they have a ratio of one gameserver per thread / core.
We generally didn't go higher than 10Hz since you'd start saturating lower end connections and any well written game will be good at reconciling state via dead-reckoning. You have to handle 100ms+ spikes anyway so it doesn't make sense to run the network super-fast.
OW just has a restricted rate option for low bandwidth connections (added when they went from 23 -> 61 Hz).
> // 60 for updaterate is LAN ONLY use 13 for internet
> // 20 is default but will cut the maxplayers you can handle in 1/2
> // for SRCDS Servers use 30 - you might be able to use 20
> // sv_maxupdaterate 60
> sv_maxupdaterate 13
Like I said, depending on how good your potential-vis code is you may be able to go higher but you risk saturating your client links and most game netcode doesn't handle congestion nicely. My info is a little out of date(2-3 years) but back then we had to keep data rates below 20kb/sec(and ideally 10kb/sec) if you wanted low jitter across the link.
[1] https://support.steampowered.com/kb_article.php?ref=5386-HMJ...
Overwatch has a 63Hz tick rate over the internet.
Every game is going to be unique and gameplay structure has a huge impact on your data rate. We had games back in '97 like Subspace[1] that handled hundreds of players in the days of dial-up and 200ms+ pings. Generally if your data rate per-link goes above 10kb/sec you'll start to see degradation of jitter across your userbase. There's smart ways to handle that but if you want to do a fixed-tick you run up against that limit really quickly.
[edit]
Other games will do smart things to keep skill high and tick rates low. CS(but not CS:Go that I know of) used to famously rewind time based on your latency so your view of the world was accurate, but lead to fun behaviors like rubberbanding as you got shot around a corner(due to movement speed slowing down at that past point in time and the game having to reconcile it with the current timestamp).
[1] http://www.eurogamer.net/articles/2016-08-12-looks-like-bliz...
Compare that to say, fighting games which are very reactionary and very hard to run over any latent connection without some seriously complicated netcode involving time rewinding.
It's also not actually why you rubberband - that's just caused by your own latency (moving while already dead). It does mean you can get hit after moving behind a wall though.
CSGO matchmaking servers (run at 64 ticks) force you to use cl_updaterate 64.
Even in CS 1.6 days, most leagues and servers enforced updaterate 100 and rate 20000. Nearly every player could handle it. And those who couldn't, well, bad luck for them. Playing with 100ms+ is no fun for anybody.
I don't know about the CS 1.6 days, I used to play a ton of early cs and was an admin on one of the most popular Frontline Force servers. We ran tick rate of 12 if I recall correctly.
Unless you had dial-up users, a tickrate of 12 is very bad and not fun. I don't know if you had hardware restrictions, but tickrate is independent of updaterate that is sent to the clients. By enforcing a low tickrate you maintain a very old snapshot of the world on your server (83 ms), which makes clients interpolate more.
I've always wondered what AWS rates do large users receive from negotiations?
I'd like to draw your attention to Planetside 2. A pretty recent FPS that was run a lot like WoW, with 3 'continents' and complete seamlessness on those continents. Firefights where 100 people were actively engaging each other were an almost constant experience back when I played it. I think there was a cap of 1000 players per continent. Those players could gather wherever they wanted. I'm pretty sure I was in some 300 player battles, and nothing but lag would prevent all 1000 people on a continent from going to the same base.
It has to be said that in the big battles (at around a 100) lag became an issue. I'm not sure whether this was clientside, serverside or both.
(There is also eve-online where recently there was a 'fight' with 5000 players in the same location, but that is a different beast from FPSes, and the main logic of that game is single-threaded)
I remember the crown frequently having 96+ per faction without any lag problems. Must have been clientside.
> (There is also eve-online where recently there was a 'fight' with 5000 players in the same location, but that is a different beast from FPSes, and the main logic of that game is single-threaded)
It was 6000, and it was terrible to be in. It took me 10 minutes to enter a station (normally takes a second) and I had to relog because I couldn't get out of the station. Relogging took another hour of black screens. It was pretty neat when I finally got out, but there wasn't much of a fight compared to previous encounters.
PS2 was for the most part an amazing game. The graphics were truly beautiful and the gameplay was very fun. Unfortunately, their shitty performance basically killed the game by making it unplayable for many users during the trademark huge battles.
I might be off on the numbers though.
> actually increases resource utilization exponentially
I think you'll have to choose there.
And we do indeed take advantage of reserved instances. We’re also looking at other options along these lines.
Quadratically actually.
Exponential would mean each user doubles the amount of calculations, for instance.
Auto-scaling instances on EC2 only makes sense if you don't have the in-house skills to troubleshoot/manage hardware servers or if the traffic is unpredictable and the bursts are infrequent (e.g. once or twice a week or just busy for a short time period of the year).
You pay for hardware that you need to mange and have to be sure that you will be using it after the game launch. ( most games player base decrease by 2x just after release. )
No, it's cheaper once you factor in the real performance of physical vs virtual machines.
Someone really needs to bring some actual numbers to this conversation.
Extraordinary claims require extraordinary evidence.
Maybe it's possible that you're wrong?
Brendan Gregg has written here and elsewhere about the performance of VMs on EC2. Worth reading if you're interested in figuring out how close it is (or isn't).
Which costs time, money and opportunity cost. Something that all the "rack and stack" people seem to forget when they compare owning bare-metal servers against managed cloud providers.
The amount of time these "bare-metal" shops waste building and maintaining infrastructure management tools, scaling tools, reporting and monitoring tools is mind boggling. None of this stuff has anything to do with the core business the company actually is in. Opportunity costs here are immense....
System goes down and it was hard resurrecting it since the traffic just kept pounding away. No autoscaling, no cloud. It woulda been handy to just fire up some more servers, let alone have things auto-scale via CPU %.
This clearly shows how complex a system is needed that has to handle 3.4 million concurrent, connected users. I think the connected part compounds any scale problems you have since it is implied they are connected to each other.
I'll never accept donations again for a game I've made available for free from the amount of problems someone has caused me after they gave me $5.
I suppose I could offer a refund, but refunding a $5 donation is just another example of how a donation box is a waste of time.
In the future I would at least consider a Patreon system where donations renew every month since server and development costs certainly do. Way too many people think they're buying an annuity against your free time with a one time donation/purchase.
It's one of those things, though, that reveals the human inclination to remember negative events more even among a sea of positive events... Frequently as a game developer you will get loads of positive and helpful feedback but the small minority that want to shit on you for no good reason stick in the front of your mind.
One thing I notice from your site is that you provide _many_ ways of giving feedback. The Discord widget, the 'Contact Us' menu item, the FAQ is peppered with communication links, there's a Forum, Wiki, et al. Basically, I'm seeing an incredibly low barrier to entry for anyone wanting to express themselves to the developers.
Have you considered paring that back? IE, ensuring all communication channels are moderated, and limiting it to only one or two services?
My team of community manager volunteers is pretty active with moderation in-game and out-of-game so it's gotten pretty good. But still, I think I need to isolate myself and the other developers from the public, because it can be really demoralizing when you spend 30-40 hours of free time per week on a side project for free and you hear some terrible things.
Anecdotally, I'm seeing more and more peers leave the industry as a result of being disgusted and worn down by the constant, unending abuse that game developers receive from users.
0: https://medium.com/@morganjaffit/the-cost-of-doing-business-...
The games as service part prevents players from keeping proper emotional distance to games by not allowing them to have closure; to look at the game, say you've beaten it, and move on. Every game needs some kind of grind or persistence beyond the actual play, through online connectivity and often actual player competition. So you kind of prime players through negative play models very often, especially when in-game purchases exist.
What's worse instead of an end to a game, devs are forced into chasing the mythical idea of balance instead. This actually sets players up into warring factions because balance is often zero sum; my warrior gets buffed at the expense of your paladin. There's no existing balance state, ever; you can look at Overwatch to see it just ending up into random changes to heroes that please and incense players in equal measure.
These things combine to make players strung out over time; their nerves are jangled through constant grinding, death (screw roguelikes!), or competition, and then you end up with developer as God who made decide to alter everything to please a particular player faction, or even no one at all but themselves. You make your players invest a hundred hours in your game, and they aren't going to be disinterested enough to take changes in perspective; you don't give them the chance to be, they have to be logged on every day doing busywork either through ranking or grinding.
I think this explains why modern gamers can be so enraged. It's more than personality; a lot of the structure of many modern games can really create this. The games make players too invested in them too easily, over too long a time.
The biggest problem here is that it largely used to be a distributed system. You used to just be able run your own dedicated server on whatever provider you liked. The dev would just run a single server list. Now many game developers have decided they're the only ones who get to run servers - primarily because they can charge more for micro transactions and private servers this way as far as I can tell.
It really hurts games - PUBG is the best example I've seen - constant lag issues, complete lack of server side checks for things like shooting through the ground (because hey, that costs CPU and every additional cloud server they need means less profit), etc. It's basically made the game unplayable.
Game developers are unfortunately stuck between immersion in their games and the rage that leaves players with when technical issues occur. The more immersed your players are, the more rage they'll experience when your game crashes or lags at the wrong time.
I believe its more about standardizing the user experience. Now that enough games have dedicated servers with a standard expectation of latency your game looks poor if your user experience is beholden to community servers with selfish moderation and varying stability.
The added benefit of centralized server hosting and control is exactly what you point out though.
The reason for banning community servers is probably mostly for combating piracy; by eliminating the ability to run on a LAN the game must always call home to be played.
It also hurts the game's image if many servers spam players with "Donate $20/mo for the MASTER rank and get exclusive weapons that deal twice the damage of normal weapons!"
I haven't played in a while but I remember playing Minecraft when I was still in school and servers were filled with this- many would let you donate hundreds for admin/OP/whatever, many would restrict core parts of the game to "donating" members (such as mining certain ores, going into certain areas, etc). Eventually it got so bad that Mojang actually had to step in and (vaguely) threaten to sue any server owner who does this - http://andrewtian.com/mojang-threatens-lawyers-against-pay-t...
Also, admins that can ban players for any reason, who are picked by the server owners (and not the devs), can result in backlash if someone influential is banned.
That's a problem with the client design. Don't allow the server to send banners or any map tiles and that problem goes away.
Community/matchmaking servers don't distribute ("run your own server") too well. If you only ever want to play with and against people you know, that's fine, but a lot of people don't want that: some want to play on randomized (or ranked, if the game supports it, across a large number of players; not just the ones in the potentially-altered rankings on a single person's server) teams. Others (more commonly) want to play against new and different teams.
Unless your graph of social connections willing to agree to use a given server is big enough and the right shape, hosting your own matchmaking/community services just results in isolated islands without a vibrant overarching community.
Put another way: the separate-server approach of e.g. TF2 works fine for some, but a lot (judging by play counts) of other folks prefer games with a centralized community.
To be clear, I think it's fine to have the option of a separate community server. But if a game developer already has the centralized infrastructure, is making tons of money off of it, and spending tremendous resources keeping it intact (as Epic apparently is), it's a pretty tough sell to say "oh and you should also spend the resources to make a hosted option for this".
"Take the complex thing designed for centralized, single-instance internal use with a team of support staff, put it in a box, and let users run it on their own hosts" is far from a simple proposition.
I'm unsure if any companies doing centralized community servers/community federation also have a large portion of their actual game servers not hosted by the company; I'd love some examples of that phenomenon. The examples of Minecraft and TF2 come to mind as doing some subparts of that, but those games only have limited matchmaking capabilities.
A whole lot better than spoon feeding customers bullshit for weeks while hamstringing your product rather than investing in it (looks at EA, mumbles about SimCity).
Because being honest can hurt the share price.
The success or failure of the recent Sim City game was hardly EA's number one concern. Fortnite, however, is probably extremely important to the folks at Epic.
I'm not defending EA here, but it was definitely in Epic's best interests not to piss off Fortnite's player base given that there are some other, ahem, similar titles on the market right now.
Epic is a pretty big operation too.
2. they actually care
3. they enjoy what they are doing
One thing I really miss working solo is working with a team of smart folks to solve a complex problem together. It's so damned fun.
(I work at Epic on Team Online, aka the team that this whole PM is about.)
They talk a lot about reducing operating complexity and scaling their infrastructure, I wonder what the cost of their current infrastructure + the staff to maintain it might be vs the managed solutions that cloud providers offer now.
For example, using cloud datastore or spanner or big table as a persistent layer, these managed services can definitely scale to the current need and I've seen them go much higher as well.
For logs ingestion and analysis, big query can be a very powerful tool as well, and with streaming inserts that data can be queried in near real time. For things that are less urgent, batch queries. For other things dataflow can help with streaming workloads.
I think one of the problems they alluded to though was that at the moment they're on a single provider, and what they're looking for is a multi cloud strategy which totally makes sense. A lot of the above products create some kind of locking, with some exceptions, like using hbase as an interface to big table or beam as an interface to dataflow. Though I don't know what the other providers offer that may have these same interfaces.
Another option is kubernetes, which I believe all providers are pretty strongly embracing. Having most of the supporting infrastructure be brought up with a few kubectl commands could help them scale across several cloud providers quickly.
Usually there is a cost/scale threshold with managed providers where it is cheaper do DIY than to pay thousands upon thousands per month for say log ingestion.
The other problem is that most of these things aren't easy to migrate to. AWS RDS is much easier because its just managed whatever you're already using. But cloudspanner? DynamoDB? You have to completely re-architect your application. Then you have to move your application, and data, to this new system...without massive outages. It's a lot of work and a lot of cost. So until things go HORRIBLY sideways, most companies don't have the spare time/money.
Been there, tried that.
At my current company we have access to AWS support. I'm not high enough on the ladder to know the specifics (separate contract, size threshold?), but when we've had issues in Aurora we've had personalized support. I have no doubt if you're scaling to thousands of instances you would have personalized support, if for no other reason that a cloud provider would not want to lose such a huge customer.
You think GCP offers better support on spanner et al when customer is having performance problems? In this case probably yes, because an Epic sized monthly spend is highly effective at escalating through support.
It takes low effort to find <cloud persistence horror story> around here so we know cloud is not a special magic that is immune from integration performance problems. But the economic incentives are meaningfully different and especially so at runtime.
I've always thought that XMPP would be useful for games, just surprised to hear that people are actually doing it.
it looks like a great business model to take something open source and hard to use and make a proprietary version thats easy to install and use
https://en.wikipedia.org/wiki/Signal_Protocol https://en.wikipedia.org/wiki/WhatsApp https://www.cisco.com/c/en/us/about/corporate-strategy-offic...
Jabber is xmpp.
Not sure about WhatsApp.
whatsapp at least was in the "early" days. they had some extensions to it but you could connect to it pretty easily (once you had a valid account generated with a real mobile number) not sure if it is still the case, things changed a lot I guess, especially with encryption.
You couldn't see friends lists at all during that time period. So you couldn't queue up in a friends / people you knew at all in a match, the only options were either playing solo or using a "filled" team with random players.
I've been playing fortnite as one of the early 60k concurrent users all the way to the 3.4M, so its been interesting seeing their load / server issues over time and then reading this (Granted, I don't understand everything discussed in their blog). They've done a outstanding job handling their growing traffic.
One thing I've noticed with Fortnite, compared to PUBG or other MMOs, is how large their patch updates are. Its usually several GB large, and it comes fairly frequently about once a week.
Most notably, whenever you wanted to add someone to your party. You had to do the following.
1. You Send user friend invite
2. User would have to accept invite
3. User would have to disconnect to refresh your friends list (due to friend-service issue mentioned in blog)
4. User would relog back on (took approximately 3 mins)
5. You could then see them on friends list
6. Send party invite
Those would be "absolutely horrible" to me. Time dilation is a compromise, not just arbitrary lore that became a game mechanic.
I'm fairly sure there isn't.
Disclaimer: Riot Games Engineer :-)
I think this is where Stacktical helps with proactively detecting performance regressions at the CI level, before they hit production: https://stacktical.com
Disclaimer: I am Stacktical's CTO
I wonder whether Epic can solve its problems by rearchitecting more into a CQRS driven system with event sourcing: store events in a more write optimized DB (e.g. Cassandra) and then process the events for fast reads through whatever is required for the usecases. Maybe they touched the limits of MongoDB to handle both, reads and writes at their scale.
[1] Getting ~16 FPS on medium settings, with the high-end late 2016 MBP 15".
Note: I did tweak some minor settings like AA to get better performance.
- Followed by removing Nginx + Memcached couple altogether out of equation.
edit: nevermind, it is available again - weird