Hacker News Reader

Comment by defertoreptar | Hacker News Reader

The author didn't benchmark to see if this system is actually any better at predicting outcomes than vanilla Elo. That's how you determine if your implied win probabilities are accurately being derived from rating differences. The author seems to be under the impression that there's something fixed and concrete about an 1800 rating, but when you change the system, you also change what an 1800 rating means in the first place.

Some of these complaints are solved by existing systems, namely Glicko. For example, rating deviation helps with experienced players (low RD) losing points to newer players (high RD). It also has a built-in way to discourage inactivity. Players' RD increase over periods of inactivity, so they can be excluded from the leaderboard after reaching a certain point. That allows us to maintain their rating without decreasing it. After all, that's our best guess of the player's skill. It's just a less reliable guess over time.

View on HN

There have been four rating systems, including Glicko and TruSkill, received lots and lots of complaints for both those systems. This new system receives few complaints. Tested across 135000 players. If the players had not complained so much, we would still be on Glicko. Those are the facts. The theories as to why that is are up to you.

Optimizing a rating system for minimal complains, maximum player engagement, or some similar metric is of course totally valid. It reminds me of Sirlin's story of being hired to design a rating system for Starcraft 2, and optimizing for totally different things that Blizzard wanted [0].

If the author (you?) had just described it in those terms, it'd be hard to object. But the article goes further, and makes claims about the system being more accurate due to a different rating curve. That's the claim that would need to be justified by actually comparing whether the predictions the new system makes really are better.

[0] http://sirlingames.squarespace.com/blog/2010/7/24/analyzing-...

Part of a matchmaking algorithm that increases user engagement is telling a story about how it's more fair though.

Like our tolerance for losing is acquired. Most normal people losing in League for the first time stop playing, usually forever. Just randomly visit your friend's match histories in League, frequent players have many days of long losing streaks.

If you're just conditioned to play despite losing, great, in a Darwinian way (surviving, being around to be measured) you will be representative of the average player in League. And there are so many League players with such long retention you cannot possibly argue that skill-based matchmaking is the core component of user engagement.

His dataset is interesting because it will necessarily overrepresent people who kept playing despite the old system. That sort of refutes its importance - I mean sure people complain but they keep playing, so was it really that important? So what if complaints go down?

Those are important goals, and also, it's still an interesting twist in multiplayer game design. You just gotta interpret it as a commentary on a whole system even if it doesn't narrowly talk about a scientific objective like performance prediction.

Player retention was significantly lower before the new rating system was introduced. It wasn't just complaints, it's just that is a more direct metric because multiple changes may have affected the retention.

The funny thing about SC2 is that the player's MMR (matchmaking rating) is decoupled from rank, due to design decisions such as demotions not occurring midseason. So a gold league player with a low enough MMR may get matched against bronze league players, despite ostensibly being ranked higher. Actually for the longest time after release, player MMR was not visible. It took 6 years and 2 expansions before it was finally displayed in-game.

I like the league system in sc2, it allows you to see progress and to get to know other people's style (because you meet the same players several times in a quick succession). People build narratives around that (I remember that guy I lost vs him last time and now I had my revange).

If somebody won against you with a particularly dirty strategy you can get back at him and you won't be fooled the next time.

It's better than dota system where you're most likely never going to play vs the same people again (or at least it feels like that).

Also the whole 5v5 game that lasts for 1 hour format is inherently very toxic - most of the time you know you lost after 10 minutes and yet you can't disconnect and have to keep playing for the next 30-50 minutes while everybody blames everybody else on your team.

In sc2 if you know you lost after 5 minutes it's discrespectful to continue playing :)

Oh, I love those guys who "know you lost after 10 minutes" in Dota...

So there's 10% chance I'm wrong. That's 10% chance of losing some mmr vs 90% chance of losing 30 minutes of my life. I don't have much time to play so in sc2 I would just go to the next game.

Sadly in a team game you can't make that choice because disconnecting too much will prevent you from playing all the fun modes (and will put you into a low priority queue where you play with people who constantly insult everybody or kill their own allies).

That's the root cause of all the toxicity in dota.

I've played Dota for way more hours than I want to admit, and unless you're playing on the highest of levels (Immortal) where people really have a very strong feeling for the win-conditions that they have and when it becomes very hard to reach them, I call BS on your statement. There are very strong reasons why pro players almost never call it GG before at least one rax is down, and often wait to do so for at least one last hail mary fight (depends a bit on the patch though).

The chance that the enemy throws a fight or makes other mistakes that opens up new win conditions (or sometimes that simply one guy in the other team tilts and throws) is just too high in Dota in order to be able to call it quits after 10 minutes.

One of the main reasons for the toxicity in Dota in my opinion is that people don't understand that there are 10 variables in the field, but they can only control one of them. Don't even bother trying to influence the other 9 (apart from positive feedback/encouragement/communication to the team), it's a waste of time and only contributes to the toxicity.

I'm at 2.5k, almost always playing as pos 5 winter wyvern. I'm under no impression that I'm great at dota :)

At this level with the same position and firstpicking ww I get a small subset of heroes I play with and against, so it's much easier to call. I have pudge and sniper every other game for example, and enemy pos 5 is very often cm or ogre :) It's a great day I get to play vs meepo or oracle but it almost never happens. The best I can hope for is axe or legion.

Also half the fancy strats don't work because people don't talk. Split-pushing is just split-feeding. Nobody does 3-lanes. Jungling is a last resort of carry that lost the lane and when he's fat the barracks are already destroyed.

So if we lost laning hard enough and have earlier-game cores than the enemy - we lost, it just takes a lot of time to play out.

But even if I was wrong 30% of the time instead of 1o% - I'd still happily take that trade in mmr if I could. Obviously pro players have different motivation so for them it's "never surrender", but almost nobody is a pro.

Not to mention that the bonus pool feature has long been removed.

More importantly, though, your MMR has now been split up across the three factions that you can play as. If your Terran gameplay is much better than your Zerg gameplay, you'll be facing higher-ranked opponents when you play Terran, than when you play Zerg.

This has made it easier for players to switch their factions up, without being punished for it.

What you said makes me wonder about a totally different way of using a metric of how likely the person is to win a given match.

That is, perhaps the system could be engineered to maintain a more even win/loss ratio so that people don't go on super-long win (or loss) streaks in general by adjusting who they get matched with.

It probably wouldn't work that well towards the edges, but around the middle it might work well enough.

This has what dota 2 does. Every win / loss is worth the same points, the games are extremely close in terms of player rank, so and it's a team game so you get more variance than just your individual skill. Players eventually settle close to 50 percent win rate

The "different rating curve" appears to be the actual historical probability data, not a formula. I think. If that's right then it this estimate of the probability of winning is not a new discovery.

On the other hand I suspect the historical data really is the "best fit" to the historical data.

You have to decide what the purpose of the rating system is. Using it as a reward system for players to feel accomplishment is a different use case then trying to correctly predict the likely outcome of a game.

Personally, if I was designing a rating system, I would use two separate systems.

One would be like the one in this game; publicly viewable, pleases the players, and gives a sense of accomplishment.

Then, I would have a second, internal only rating system that players can't see but is used for matchmaking to make sure people are matched up to players with as close to equivalent skill as possible.

I find similar solutions experientially two-faced and frustrating. Imagine the matchmaking engine was a person behind a desk that you interact with. If he consistently told you one thing and then did something else you'd be displeased.

Some games do this already, and players are very unpleased by the results.

Probably because they get matched to tougher opponents if they're better?

It's kinda disheartening being matched to plat players in your silver promos.

That's what xp is in sc2. For several years xp was the only number that was visible - your true mmr was hidden.

Peple knew this and ignored xp altogether.

My understanding was that the system consists of using the historical odds of winning (given the rating difference). If you benchmark that using only past data, I think it is by definition the most accurate system. (The data is always a better fit to itself than a theoretical fit is.)

Naturally future data is much harder to deal with than past data. But even for future data it's not obvious that ELO (or any other theoretical fit to the odds of winning) will be more accurate than the historical odds.

Yes, the best fit for the data is the data itself, it's a tautology. Nothing wrong with Elo's exponential curve, it just can't beat the actual data.

You raise a good point in that I could've created a training set and a test set, that probably would be a better validation. But I don't know, I'm not doing science, I'm making a game.

On the topic of whether the future matches the past, the predictions were based on a rolling database of the past 100000 matches, which is approximately the number of matches played per 7 days. So my theory is that the data is quite recent and up-to-date and so should match, in general.

Of course I never tested this. In the end, I'm not doing science, I'm making a game. If the retention goes up, complaints are down, then I can't keep working on the rating system, there are 1000 other things to do.

Yeah, I'm not giving advice on how you should do it. I was just unsure whether critics here had understood that measured data is probably better than any theoretical fit, even the revered ELO.

> I think it is by definition the most accurate system

By gum, an opportunity to quibble semantics on the internet. That is true if benchmark using means 'only admit to knowing' and accuracy means 'must be numerically quantifiable given existing data'. It is false otherwise, especially if accuracy means 'conforming to truth' and we have a model for how the numbers are being generated.

Obviously if I generate a set of numbers by sampling a normal distribution then the most accurate model is a normal distribution, no matter what empirical data I use for benchmarking.

That is to say, if we know how the data was generated (sans noise) we can reject empirical distributions as the most accurate, because we can directly know the distribution of the data.

Ok, that is a legitimate ... quibble. Let's assume that we don't already know the correct distribution. In that case we're going to judge each theoretical fit by how close it comes to the historical data. (Or else we're going to get that wrong, which is another common approach.) ELO is much more prestigious and credible than some guy who made a game, but it is less credible than data, for some number of data points N. (Although I think a theory can be more prestigious than data almost independent of N.)

Well, it’s like the question of what is better: a restaurant with 4.5 stars on 4 reviews or one with 4.2 stars on 1,500 reviews?

Sure. If there's enough data then the data becomes more credible than even the most popular theoretical fit. If I have four games I played with my nephew then people should probably go with ELO.

Is the point really predicting outcomes? FIDE (chess) Elo is useful because I can compare machines to humans who have never matched each other.

Generally speaking the "rating structure" is a lattice where you can, for any two players A and B, tell whether A is a better player than B or the other way around. Elo, Glicko, etc. are embeddings of this lattice on the real line (much like the utility functions of microeconomics are real embeddings of preference lattices).

> Generally speaking the "rating structure" is a lattice where you can, for any two players A and B, tell whether A is a better player than B or the other way around.

Not really. You tend to have cycles, where person A can beat person B who beats person C who beats person A.

There's a guy who was even with the guys playing Go who were 2 or 3 stones weaker than me that would tend to beat me because of some of the unorthodox things he did. (Eventually I strengthened my game against these things).

Considering ratings to be a total ordering is a useful approximation.

> Is the point really predicting outcomes?

Others have pointed out how there is a psychological aspect of rating systems, and no developer wants to constantly field complaints. That said, I believe the answer is yes. A rating system derives meaningfulness from its predictive power. In other words, people want to know how good they actually are compared to one another.

I think that for most game players, outside of the top-N group who just want to be at the head of the list, rankings are largely a mechanism to facilitate playing good games, where good is generally defined as close games where both players feel like they could have won.

There's an interesting question about how you rate players who use fundamentally different strategies. For instance in RTS games, should you match boom vs blitz players of otherwise equivalent elo? Or should you instead (try to) construct a classifier to determine which type of player someone is, and then have a rank against each other type of player and match them according to that rank?

As long as you let players play several games vs each other it's fine. If you met this cannon rushing guy already you'll know what to expect.

That's why people use barcode names (llll1l1l1l1l111l) in starcraft :)

For a lot of online games I think matchmaking based on pushing you towards a 50/50 win rate is kind missing the point of games. It gives you fair odds of winning, but it doesn’t necessarily give you even odds of having a fun or competitive game.

At high skill levels players are skilled enough to where it might, but with most online multiplayers the overwhelmingly vast majority of players are lacking in basic fundamentals to varying degrees. At that level, ELO based matchmaking mostly just results in one person getting rolled or doing the rolling. They’re not really competitive games in my experience.

If two players of similar skill general roll one another, that's a game design problem not a rankings problem.

Reply on news.ycombinator.com