League of Legends Chat Service Architecture
engineering.riotgames.com
engineering.riotgames.com
I really wish more game studios would do this (Blizzard especially).
http://www.riotgames.com/articles/20150817/1697/our-engineer...
They also mention that their users warned them about this battle before hand which was really cool.
So, I know that I'm expecting too much, but remember that the only way CCP can handle large fights is to simulate them at 1/10th realtime, and get advance notice of said fights so that they can move the simulation for the target system to a single, beefy server ahead of time.
It's a goddamn shame that they don't have the resources available to figure out the -admittedly difficult- problem that is
* Reworking their simulation to be parallelizable across nodes
* Dynamically shifting slices of a battle within a system across physical nodes
"Scaling League of Legends Chat to 70 million Players" by Michal Ptaszek https://www.youtube.com/watch?v=_jsMpmWaq7I
If there was a language that let me define something in terms of an individual actor-modelled components, but then allowed you to optimize some of them into batched GPGPU passes (and apply strict simulation ticking and so forth in the process), I feel like it would "win" all sorts of domains. Erlang could be that language with a little work.
Sorry to veer off topic a bit, but as a Python-lover-but-outsider I've long been mystified by the almost proud unwillingness of the community to embrace Python 3. It seems like a better language to me. What am I missing?
1)The "latest version" of a programming language is not the same as having the latest version of Firefox. Having the latest version can actually be a detriment if you lose libraries as with Python3.
2)It may be unpopular, but I tend to follow the market to a certain degree. Python is my language of choice for home projects and employment. But almost all employers use Python2.x. So I follow them just to keep 1 less hassle out of my life.
3)Losing the long tail of libraries that 2.x has. Many things support Python3, and I'm not against it. I'm just waiting for the day when Python3 is more convenient than Python2 to move. I wasn't one asking for the Py3 changes, and they certainly weren't done for me. So I don't feel like being inconvenienced or doing any porting/heavy lifting in the meantime.
4)My petty reason. I think everyone should just do what's in their best interests. I advertised that "2 of course" because it is a given, but rarely stated. I've met a number of pushy Python3 folks and find them distasteful, which hasn't helped budge me in the slightest. They've probably made me a little edgy on that topic.
5)It's a programming language, so what helps you build something the fastest should be what we use. Today that's still Python2. Some people feel very passionately about Python3 and I hope they spend many hours porting and building the ecosystem for my arrival one day. What seems to be going on instead is they're hoping to convert more devotees that will go out there and do a bunch of free ports.
6)I'm more concerned with moving too early than I am too late when the Python3 ecosystem one day, maybe, eclipses Python2. They're just not that different, I've used both and it's no different from building software with 2. But they have vastly different breadth of libraries in 2's favor. They also screwed up on a few things with 3, it isn't really a flat "upgrade" either.
7)Python3 is just adding more bloat. I generally prefer features are REMOVED from a language until there's nothing left that could reasonably be removed from the core language. Ideally I'd like to see a small version of Python with the basics never touched and features added through addons or a macro system.
Latest version, upgrade, and support-ends-2020 are all political statements in regards to Python3 that mean nothing in reality. Latest version- what can't be backported? Upgrade- more like sidegrade once all things are considered. Support EOL- total smoke and mirrors, code will not stop running post 2020.
They're tossing every feature against the wall that they can to fix that. I'm one that holds feature-soup in disdain though so this is unattractive to me, and I prefer a well thought out "one way to do it" (Zen of Python) approach. Python is well beyond the basics at this point and not a small language, but at least 2.7 is frozen and I can choose to use the backports people provide if I want. I like a smaller core language if possible because I have to read other people's codebases.
To me something like Firebase could work for this?
It seems like when game dev companies grow they love to blow out all of their own in house solutions so they can have complete ownership and control. But is the cost of ownership of your stack really worth the maintenance costs? Why not outsource as much of the stack to another company that can take advantage of economies of scale?
If you're doing more than 20k concurrent users, odds are you can afford to hire someone to scale up or build out a team to do it, or find a 3rd party service and pay them to host your chat.
Openfire is a known server for XMPP:
Yes it is, and so is ejabberd, in GP's comment. (And the most popular iirc)
They were talking about indie game studies, not indie chat protocols.
ejabberd is an XMPP server written in erlang. As soon as I saw they talked about erlang I was pretty confident that ejabberd was the next name they would drop. Very surprised they suggest they rolled their own.
After working with (hacking around) OpenFire for two years, I 100% agree with your statement. The clustering plugin routinely fails (to the point where we have actually investigated not even clustering it anymore), the admin interface routinely displays "wrong" data, and other fun bugs we found along the way.
Does it work for us (with a lot of client XEPs and additional custom plugins)? Yes, it does an acceptable job. Would I choose the same product again, if given the choice? No.
Dota 2 is churning along quite comfortably with integrated voice chat despite having more inter-region mixing. Any time this subject comes up the Dota 2 community nearly unanimously agrees that voice chat in no way encourages "toxicity" and more likely actually prevents it by assigning a human voice to the player as opposed to just disembodied words.
EDIT:
Are there any articles like this on the architecture of Dota 2 or Steam?
Although the mute system is still a must have.
No one was using it because it wasn't as good or easy to use as already existing third party programs, so they dropped it. Riot is probably thinking the same, and can spend their resources elsewhere on something that isn't already done better by someone else.
Contrast this to LoL where you're dropped into a game randomly with others random players for <60 minutes who you don't know but are forced to communicate with to maximize strategy. Communication is utterly critical yet the best option for doing so is slowly typing a message in a micro intensive game or a nondescript ping. The resulting player behavior is exactly as one would expect.
Players aren't consistently using voice chat since it's not built in. If you are in a match up where one team is on voice chat and the other is not; the team with voice has a huge advantage as text chat is less visible and has more delay.
If anything it makes the game better when people start using voice. I've had games that I've given up on and people started using voice to ask for things like shared unit control for dominated creeps to stack ancients and actually communicate where they're going instead of trying to guess what's going on in four silent teammates' minds.
OTOH, I had one game on EU W where two Russians were talking (nicely) to each other and some British guy comes in and just says "I can't stand Russians. Good bye." and leaves.
And I've had my fair share of LAN cafe players with an open mic blaring music and their friends conversations. Mute and move on.
I really do wish we could mute the mic and not chat and vice-versa.
There's so much talk about toxicity in league, and very little about what CAUSES the toxicity.
Hands down the biggest issue is not coming down quicker on trolls and AFK's in ranked. They have a system in which they literally FORCE you to play out an hour long game when someone never connects, leaves after 5 minutes, doesn't get what they want so intentionally feed, and so forth.
For someone trying to climb that's automatically 2 hours gone. 1 hour for the game they're forced to play out, and 1 hour for the game they have to win to even out their ranking loss.
I have literally seen streaks of 60-70% of games with one of the above (troll or AFK). Eventually people get frustrated by it and that's when the toxicity starts.
If Riot wants to improve toxicity they need to protect the ranked experience better. I have so many things going on in my life that losing 2 hours because some asshat on the internet decided to troll is just not worth the time. 2 hours is a LOT of time, and while I loved the competitive nature of the game, eventually I just walked away from the game.
Shortly after I decided to walk away from the game, I came across the following video, and it describes EXACTLY why I stopped playing, and where my frustrations with the game where.
https://www.youtube.com/watch?v=5UfO7qOHx5M
The day I quit was when I sat down on a Saturday to spend playing league (one of those rare days where I got to do what I wanted). I spent 2 hours waiting on dodges (where it makes you wait when you leave during champ select). I kept dodging because I kept getting trolls in champ select. And then something like 4/5 of the games I did get into had either an AFK or someone who got pissed off in game at someone and started trolling.
I just thought to myself, what am I doing? There are so many more productive things I could have done today, this game does nothing but frustrate me.
I haven't played a game since, although I still watch LCS.
If you abandon 2 ranked games you get put into the low priority queue and have to play a certain number (at least 5 I think) of all-random games that don't record anything.
If you get reported for communications abuse enough you get a communications ban and can only use the chat wheel and rate-limited pings. Other reports get turned into low-priority queue time (supposedly).
If you dodge a game (even if you don't ready up) you get progressively harsher queue timeouts. It goes something like 4 min, 10 min, 30 min, 1 hour, etc.
There is also rumor of a hidden queue pool of toxic players. Kinda like a shadow ban but it lumps all the ragers and dodgers and AFKers and feeders together to play with each other.
Honestly, LoL never appealed to me. Every champ is about the same with about the same skillset and every game has the same laning. Riot seems to not really be able to make changes and their community management is bad.
If you can get past the turn rate and inability to spam spells in Dota you might enjoy Dota more.
The LoL community has this idea that dodging is an important part of climbing BECAUSE of how often you auto-lose games from circumstances outside of your control. And they're right, but only because of how badly Riot protects the ranked experience. I personally hate dodging, but I hate wasting an hour or two due to trolls and AFK's even more.
It kills me to watch riot spend so many resources trying to "fix toxicity" when the real problem is how they give other players so much control over your time in league.
I played 2 games of Dota last night. One 35 min (ranked) and one 43 min game (unranked). It took less than 5 minutes to find each game. I had one game fail to start due to fail to two players failing to load. Reborn has been having some server issues so it's not unheard of. Total time in game was like 1:30.
The ranked game was a complete stomp - 59-14 in 35 min but the losing team stuck it out for the entire game. We could have ended it 10 min earlier but we didn't for whatever reason.
No rage quits or gg ff calls or intentional feeding. I'm not highly ranked (no clue what the equivalent is in LoL; I'm just at or slightly above average) but it's rare to get RQs in ranked at my level.
Knowing that I can't trust another player to launch a skill shot attack on target is valuable info - it changes the dynamic of how you'd play with that person.
I totally understand where you're coming from.
However, there exist people who play MOBAS who are easily frustrated and -emotionally- twelve years old. Hiding the rank and ping of all players gives such people two fewer things to get angry or upset about. If they're not angry or upset, they might just either play well and win the game, or -barring that- be a pleasant person to play with. :)
> Knowing that I can't trust another player to launch a skill shot attack on target [because of high ping] is valuable info...
Back in the day, I used to play Descent and Descent 2 [0] over both a 28.8 and -later- a 56k analog modem. My ping times varied from 300ms to 700ms. There were a few weapons in that game that were -effectively- skill shots and tricky to land, but -if you were aware of your ping, and your opponent's likely motion-, landing the shots was totally doable.
In short, just because your teammate is a HPB doesn't mean that he can't cope with that ping. He might always play with that ping, and -thus- be completely accustomed to it.
[0] Both games are 3D, 6DoF shooters.
With LoL being as team-based as it is, knowing if/when my Blitzcrank is going to be able to land his rocket grab lets me know when to launch my ultimate and the rest of the team to join the fight. Having a high ping guy initiate a teamfight is.. unwise.
But quite honestly, I don't believe that hypothesis. If someone is going to get bothered to the point of toxicity by ping, something that is almost completely out of the player's control, they're going to get just as bothered being killed by Malzahar's global ultimate, someone picking the wrong team composition, not getting the lane they want, someone speaking a language other than perfect english, and so on.
It's like instead of optimizing for players to be able to play their best and give them the tools to do so, they optimize for, and drag everyone else down to the level of the emotional child. As someone who is not an emotional child (at least I think not): that's disheartening.
I also don't believe Riot is serious about dealing with the problem - it takes a long history of being a serious jerk before the bans start getting trotted out (talk of "correction cards" aside).
And it makes sense why - banned people don't buy champions and skins.
Descent 3? Yes.
Descent 1 and 2? No. Absolutely not. D1 and D2 were written for play on IPX/SPX LANs. All latency compensation was done manually by the player. It wasn't until the IPX/SPX to TCP/IP bridges known as Khan and Kali came out that folks could play them on the Internet using their shitty analog modems. In fact, a pretty big part of Descent 3's marketing was that it was optimized for play over a analog modem connection to the Internet. [0][1] (Source: I've -sadly- put far more hours into Descent 1 and 2 than probably any other video game. I could fly the first five or ten mines in both installments backwards, with rear-view off and not hit a wall. I also racked up far more in per-minute charges to my dial-up ISP than I ever want to think about.)
Edit: As I was thinking about this, I remembered that both D1 and D2 also offered null-modem (AKA crossover serial) and analog modem options for head-to-head multiplayer. Note that neither of these options transmit data over IP; the analog modem option just acted as a really long null-modem cable. ;) The analog modem option was likely added because once you've written the code to handle null-modem connections, you're at least 90% of the way to handling analog modem connections. But -again-, latency compensation was entirely handled by the player.
> ...players report games becoming difficult around 100ms and nigh unplayable >=300ms...
Here's the thing. If you have -for a little while- a > 300ms ping spike, it's going to disrupt your play. If you always have a > 300ms ping, two things are true:
1) It's the only way you've ever played, so your reaction times are going to tuned to that latency.
2) You're never going to have any basis for comparison, so (and this is key) you're not going to complain about your ping on forums.
> If someone is going to get bothered to the point of toxicity by ping, ... they're going to get just as bothered being killed by Malzahar's global ultimate, [etc.]
I generally agree with this. However, if one can easily remove two barriers to a player's enjoyment and not substantially impact the game's character or mechanics, why shouldn't one do such a thing? The action obviously produces more happiness than it eliminates.
> It's like instead of optimizing for players to be able to play their best and give them the tools to do so, they optimize for, and drag everyone else down to the level of the emotional child.
This is hyperbole. You can't seriously believe that the removal of player level and ping information from the match loading screen is "drag[ging] everyone else down to the level of the emotional child".
> I also don't believe Riot is serious about dealing with the problem...
I played LoL for a looong time. I played before the Tribunal was even a thing. I can't speak to how serious Riot is about improving the community, but the difference between the pre-Tribunal community and the community today is night and day. The improvement is vast, and has largely come through the steps that Riot has taken to remind folks that
* This is just a game.
* Your teammates are people, too.
* People who enjoy themselves are more likely to play well. People who play well win more often.
and through steps to identify and prevent chronically toxic players from interacting with non-toxic players. Maybe the isolation machinery doesn't work quickly enough for your liking, but it is there, it does work, and it has had a real, positive effect on the character of the playerbase.
> ...banned people don't buy champions and skins.
Non-toxic players who refuse to play the game because the playerbase is full of toxic players also don't buy champs and skins, and they don't introduce their friends to the game. LoL is bigger than WoW. [2] Riot gains more by permabanning chronically toxic players than they lose in lost potential revenue from said players.
[0] https://www.youtube.com/watch?v=H0aXovYR9iQ
[1] It's more-or-less illegible, but there's a section in the middle of the retail D3 box that mentions how the game is optimized for modem Internet play: http://bubbalou.event3d.com/descent/dvd/d3web.jpg
[2] http://www.forbes.com/sites/insertcoin/2014/01/27/riots-leag...
But league, I was more referring to the standpoint of everyone else on the team, not the one guy with high ping. The one guy is living in a world that's offset a quarter to a half of a second behind everyone else - he can't play better than his ping. 350ms is enough time to blink, dodge a skillshot, and so on. In casual games or lower tier ranked that might not matter or even be noticeable, but get up into Diamond/Challenger? Painful.
What if one's enjoyment of the game is based on their performance and competition (i.e. the people who play in the higher levels of ranked?), rather than vague "happiness"?
I've played enough of the game to know that being stuck in a losing game (yes yes, it's just a game, not the point) sucks. It's awful and inescapable (since quitting hurts your team and penalizes you), and a lot of the time, it happens because of factors completely out of your control, whether that be internet connections, that one guy who just disconnected leaving you down a guy, or that one guy who uses phrases ending in "...or I feed". You enter this system to compete with others, and you're denied the one thing you came there to do (and knowing how long matches can take, perhaps the only time you had to do).
Competition as a system necessarily involves unhappiness on the part of one party.. I know few people who enjoy losing.
>You can't seriously believe that..
At the risk of sounding overly snarky, you just said that riot's goal is to maximize happiness. That means removing things used by children to cry about, even if the adults have a legitimate reason for their inclusion. That is the very definition of optimizing to the lowest common denominator.
>It is there, it does work..
And yet the new player experience is still awful. I can say this, because as far as League's system is concerned, I am a new player. There's a reason some high 90s percentage of my games are with friends, rather than matchmaking games that actually raise my standings- dealing with the community sucks. People who leave, people who feed, people who just act like jerks.
If there is improvement, it's completely invisible to me. Perhaps the higher ranks have been cleaned up, but the bottom of the pyramid is still godawful.
We'll never know for sure. There have been many times where I'll type in chat: "top", my message will appear, and then somebody else will say "top" immediately after. But, a asking a third person reveals that the second player's message actually arrived first, even though my client shows my message first.
The chat architecture is terrible.
I and many people I know don't agree with this. Instant feedback is nice, but not required. Moreover, if we need to choose between (instant feedback) and (knowledge of server receipt of a message and knowledge of message ordering), we'll choose the latter every time. (I'm glaring at you, Google Hangouts.)
Having the chat server lie to you about the state of the world always feels far worse than a delay between the time you submit your message and the time it appears.
Even if the server corrected the order messages were received it'd feel shitty to not get your lane because someone has a faster roundtrip to the chat server.
2) The case where the server displays one message ordering to one player and another message ordering to another player is strictly worse than the case where the server takes 250ms longer to display the same message ordering to all players. The first case creates confusion and arguments, as well as feelings of distrust directed towards the chat software. The second case might cause a vague sense of displeasure at the quarter-second delay.
> Maybe the best way to claim top lane shouldn't be the first person to type "top" and hit enter.
Agreed. Perhaps people could be civil when they play their games. :)
[0] In this case, RTT doesn't matter. Only the time required to get a packet from you to the server matters. I think that we can reasonably assume that the worst case RTT from anyone who's playing a vaguely-latency-sensitive online video game to anywhere else on the Internet is 500ms.
[1] Honestly, it might take half of this time for the pixel issued by a draw command to your graphics card to appear on screen. [2]
[2] http://superuser.com/questions/419070/transatlantic-ping-fas...
Basically, it triggers an annoying O(N) operation whenever it happens :)
Although that matchmaking change where you get to say where you want to go and it matches you with a team helped too.
I fired them because I'm a mature professional who can figure out how best to utilize my time, TYVM.
Find a company that people actually enjoy coming to work and collaborating in. I mean, if you're an engineer, you're already doing the #1 job in the country, why poison the well at a company with a mistrustful IT policy?
Sorry but I have no end of disrespect for that kind of bullshit. If you're at a job and you're jerking around, people will eventually figure it out regardless of stupid content filtering. And in this case, this is a lost learning experience for you, as the article is very interesting.
In case anyone's interested, we just posted new content this week - an overview of our RFC process, and the continuation of a Docker&Jenkins tutorial: http://engineering.riotgames.com
http://help.twitch.tv/customer/portal/articles/1302780-twitc...
Are there any great examples of IRC being used for the backend of a games chat system?
How would you envision using IRC in this case?
Would have been great to see RiotGames write their own custom framework to interface between their game and an IRC backend, open source it for other companies to contribute to and prevent further reinventing of the wheel.
* IRC doesn't easily meet their extensibility requirements. Adding a new field to an existing IRC message just isn't doable.
* IRC has no concept of contacts. Their are bots on some servers and some clients will whois a set list of nicks on join to emulate this feature, but having contacts as a real concept is obviously preferable for Riot's use case.
* XMPP is also a standard protocol. If ejabberd didn't meet their needs, it's hard to see any standard ircd meeting them either.