Implementing the Elo Rating System (2020) | Hacker News Reader

Implementing the Elo Rating System (2020)

mattmazzola.medium.com

119 pointspmontra83 comments

One of my favorite Elo bugs I came across was a bug in a game called age of empires 2: DE.

Whenever my friend and I played online together, we would sporadically get absolutely creamed. We started looking up the stats of the others players we were matched with and noticed that on those bad games the average Elo of our opponents would jump to >2k Elo vs our barely 1k.

Both of us being software engineers, eventually the conversation turned to “What the hell do you think is going on here? How does a matchmaking bug like this crop up?”

Eventually we came to the conclusion that they must be summing out Elo for the Elo of our matchmaking party rather than averaging it.

Eventually the issue was patched and our worst suspicions were confirmed that indeed multiplayer party Elos were summed instead of averaged.

The sentiment you're expressing is the idea that one 2k player is so much better than two 1k rated players that it doesn't make sense to simply sum up each team's rating and keep it even, which is the false assumption that the AoE 2 devs made. Elo is not linear.

However I don't think they'd have fixed it just by averaging the Elo ratings, they must have implemented something more complex, like a proximity based algorithm, otherwise if the teams were the same size you'd hit a similar problem with an average as with a sum. A 2k player + a 0 rated player would sum to 2k and average to 1k. You and your friend would also sum to 2k and average to 1k, again assuming even team sizes.

That said, still very funny!

Also, fun fact: Elo isn't an acronym/initialism, it's someone's name, Arpad Elo (the inventor).

Also also, I don't think Elo is really a good idea for multiplayer (more than 1v1 that is) games generally. Don't know enough to say that authoritatively, but using it for something like AoE 2 is likely prone to all kinds of strange unexpected outcomes.

"The sentiment you're expressing is the idea that one 2k player is so much better than two 1k rated players that it doesn't make sense to simply sum up each team's rating and keep it even, which is the false assumption that the AoE 2 devs made. Elo is not linear."

It's even better than that. Elo is purely relative. The difference in score between two opponents is what is supposed to predict the likelihood of victory for the two. This means that you can add any constant you like to the entire Elo pool without affecting the Elo rankings at all.

As a matter of aesthetics, when initializing an Elo pool it is nice to try to start with values that will result in nobody having a negative value, because that gives humans the feelbadz, but that's not mathematically necessary; you can run the whole thing deep in the negative billions if that floats your boat.

The Elo math is very much based on two players playing head-to-head. It would be completely broken for almost any other scenario, as multiplayer introduces new degrees of freedom that Elo has nowhere to put. By that I mean that there may be one game for which the team's skill is essentially the max of the members, another for which is it essentially the minimum, and a wide variety of things in between. Elo has nowhere to put that "wide variety", and it was never designed or intended to do so, so that's not a criticism of it. It just isn't what it was for.

Overwatch implements an Elo system for 6v6 competitive multiplayer. Over the years they've had to make a lot of tweaks to make it kind of work. For example groups are rated by average Elo, but the system doesn't allow you to cause large Elo differences within a team. Also it differentiates between "premade" groups and groups assembled by matchmaking, since the former are assumed to be more coordinated. It also tries to be really smart about how Elo points get distributed within the team on win/loss (which causes a heap of other problems).

Overall you can make it somewhat work, but Elo really isn't well suited for anything larger than 1v1

It's worth noting that the general models ELO approximates, called item response theory (IRT) models, could handle larger than 1v1.

Its similar to how you model situations where multiple latent skills contribute to performance (multidimensional IRT). In practice, it's a way easier though to use simple stuff like averaging / constraining groups so everyone has roughly the same skill level.

I'm not really sure anyone's come up with a better system though for a game like OW.

At least in comp I don't get rolled too often, so it sort of works.

Seems to me that you need to use the max of the team rather than the average. The better player can tell the less good player to do, up to a point.

I'd expect "geometric average" to be one of the closest simple statistics for most games, assuming that you have some ranking that can be treated linearly.

It would be generally unclear that rankings can be treated linearly. A subtle aspect of Elo is that as the disparity between two players increases, the Elo metric essentially becomes less and less certain that there is any relationship between the skill of the players at all, which is probably a bit deeper way of understanding "why" . Eventually you'll reach a point where if you round to integers, the winner winning will result in an increase of 0 points. (In fact one thing to watch out for is how you round; it's easy to end up with a wandering Elo basis if you truncate both sides rather than rounding properly, for instance.)

It's fairly trivial to show that if you take a pool of the worst players and the best players, Elo will eventually converge to those two pools having about that amount of distance between them, but by inserting a fresh pool of middling players into the pool, the distance between the best & the worst will increase despite their skill not changing. If skill is linearly distributed... assuming you can even define that... Elo ratings might be somewhat reasonably treated linearly, but skill is very unlikely to be linearly distributed.

Basically, I like Elo when used properly and I have used it to good effect a few times myself, but it is a bit dangerous in that it offers a number... but that number lacks many properties we associate with integers. (For instance, in another posting I point out you can shift the entire Elo pool by any constant you like without changing the system. Integers do not have this property.) They have a definite meaning, but it isn't what your intuition might suggest. You really shouldn't do any arithmetic on them. They only make sense in the context of the Elo computation itself.

Although highly dependent on the game itself, I did some investigation back in 2008 into testing various Elo modifications on a pretty large dataset of individual players when playing in a 5v5 team game, and geometric mean of individual ratings was the best simple predictor for which team would win (the update method was essentially 10 individual matchups of each player vs the aggregate rating of the enemy team).

This is what rocket league uses. While it can be unpleasant as a high ranked player to jump in a lobby with your much less skilled friend, overall the matchmaking system and equality of matches is in my opinion utterly fantastic

Average just causes other issues. Teams should probably have their own rating not derived from the individual elo.

But honestly, elo is a shit system for fun matchmaking anyhow.

Why do you say it's shit ?

Once you reach the top there's much less reason to keep playing. It doesn't have a great way to handle multiplayer situations. It doesn't handle confidence as well as some other systems ie it takes a while to skill test a player.

https://en.wikipedia.org/wiki/Elo_rating_system#Practical_is...

Trueskill and many others are often used instead.

https://en.wikipedia.org/wiki/TrueSkill

The biggest issue is fun is not the same fair and its a lot more complicated to tune for fun.

You could, for example, play 100 matches where you were given the closest opponent but at a disadvantage for every single match. There's a lot of other scenarios around how you want to fold in ping based match making and how that smaller pool effects the fairness of the matchhing.

Elo is not a matchmaking system, Elo is a rating system.

Elo tells you how skilled a player is, how surprising a result (win, loss, draw if applicable) is, and how a player's rating should change based on a result and how surprising the result was. It does NOT tell you who should be matched up against who.

> The biggest issue is fun is not the same fair and its a lot more complicated to tune for fun.

> You could, for example, play 100 matches where you were given the closest opponent but at a disadvantage for every single match. There's a lot of other scenarios around how you want to fold in ping based match making and how that smaller pool effects the fairness of the matchhing.

Elo tells the system that if you play 100 games at a disadvantage, your rating should not go down as much when you lose, and should jump up more on a win. Elo is the math that tells the system that if your skill does not change, your score does not change, regardless of whether you play 100 games at disadvantage and lose most of them, or play 100 games at an advantage and win most of them.

I agree that Elo handles uncertainty poorly; when you're rating a chess player whose career is tracked over the course of decades, this isn't an issue, but online play with transient accounts, players who blip into a game when it's hot and then never play it again, smurfs and account buyers, people who take a break for a while and come back, patches which might radically change the meta, etc, uncertainty should be accounted for. Note that TrueSkill is, for all intents and purposes, just Elo+Bayesian inference. Like Elo, it is a rating system, not a matchmaking system. TrueSkill does not handle the problem of playing 100 games in a row against players who are better than you. TrueSkill, like Elo, just tells you what a result means, not which players should be pitted against each other.

Elo-like systems (or various other rating systems) are often as part of a matchmaking algorithm - since you can easily calculate expected game win percentages for each party, and this allows you to create more enjoyable matchups. An example of this happens even in online chess: you're much more likely to be paired against a player with a similar rating, because you're much more likely to enjoy a match against someone similarly skilled.

Agreed. Elo is fine for ranking. The issue is its used to match make and it isn't great for that.

What's the open source alternative to TrueSkill? It seems really interesting but it's patented by Microsoft.

I wrote this library: https://www.npmjs.com/package/openskill

It's open-license, handles complex matchups (e.g. 5 vs. 3 vs. 4), comparable in predictive accuracy, and is up to 20x faster than TrueSkill. The implementation is also much simpler.

This is excellent. I love how simple the API is.

They can't patent the general idea, so just base something off the information in the Wikipedia article. At time of writing, I don't think doing that would infringe their patent.

Is your project non-commercial? You can use this[1] official bsd-licensed[2] library for non-commercial projects.

[1] https://trueskill.org/

[2] https://github.com/sublee/trueskill/blob/master/LICENSE

If it's BSD licensed, how do they limit it to non-commercial use?

BSD doesn’t have a patent grant, so it would still be protected by patent, even if there’s no copyright restriction on commercial entities using the code.

Glicko

Glicko (1 and 2) don't handle complex matchups well (as in, anything that's not 1v1). Trueskill was specifically designed to extend the Bayesian concept of Glicko in a more general way which allowed this type of use-case.

I accept your premise and the fact that matchmaking is hard but:

> You could, for example, play 100 matches where you were given the closest opponent but at a disadvantage for every single match.

not really, you'd derank aggressively and end up playing people at a lower level and presumably winning comfortably after some time.

It all depends on who happens to be playing the game to match with but you're right. Ideally you would quickly hit a good match and win.

One issue is that once you find your level, Elo should keep you around a 50/50 win rate. So it doesn’t feel like you’re making progress or improving even if in a real sense you are.

It’s also a bad fit for team games. But ranking individual contribution to a team is hard. Even for something like the NBA where there’s obviously a ton of incentive to do this, statistical methods rely on a large sample size and often need to be taken with a grain of salt.

Most multiplayer games I know of, such as StarCraft, use a different score for single player and multiplayer games. You might have an amazing single player Elo score but a crappy multiplayer score and vice-versa.

That's what they do in Starcraft 2 with MMR.

Why is it shit and what system is better?

> Eventually we came to the conclusion that they must be summing out Elo for the Elo of our matchmaking party rather than summing it.

I think I'm missing something. Could you rephrase this?

*rather than averaging it, I assume.

Yes. Thank you. *Averaging. Thought I caught all the mistakes.

How would that make a difference? Or were the party sizes different between you and your opponents?

Players A and B are friends in real life and play a game together. Player A has a rating of 900 and Player B has a rating of 1100. They queue up as a team for a 2v2 match.

Players C and D and randos on the internet and solo queue for whatever. Player C has a rating of 1900 and Player D has a rating of 2100.

The matchmaking system tries to find a match for Players A and B to play against. Their Team Rating is 900+1100 = 2000, so it looks for another team rated around 2000 to play against. The matchmaking system finds a team rated 1000, and that's no good, and a team rated 3000, and that's no good either. It can't find a team rated 2000, so it goes to the fallback plan. It just tosses the team into the solo queue and lets the solo queue matchmaking system handle it.

The solo queue matchmaking system has a player with a rating of 1900, a player with a rating of 2100, and a ~~team~~ player with a rating of 2000. Perfect! It matches the ~~team~~ player with a 2000 rating against the 1900 and the 2100, which averages out to 2000.

So our opponents were queueing as individuals and were highly skilled. So their “party Elo” was genuinely 2k. Parties would be selected to take place in a match (2v2 / 3v3 / 4v4) of similar party Elos.

We would either get paired with another couple 2k players against a full team of 2k (4v4 match) or in the worst case a 2v2 against 2k players.

In the 2v2 case, because of the Elo difference the opposing side was expected to win >99.99% of the time.

Since there are an equal number of players on both teams, why would it matter if the system used the average or the sum of the players' Elo? Wouldn't both methods have the same outcome?

Two players of similar skill would probably get creamed by a team with a large skill disparity.

I guess, but (making up numbers here) two players with 750 skill would be 1500 or an average of 750.

A sum might match them up against two players with 1300 skill & 200 skill due to the 1500 sum. OTOH, the average would match them up against... an average of 750, so the same people?

I'm sure I'm just missing something really fundamental about how this all works.

They shouldn't be summing or averaging. They should be using the max elo of the players in your party for every member of your party.

I also noticed it when I play AoE: 2 DE as I only play multiplayer game. I initially thought it's just because there aren't enough players so they match us with someone else much higher. However I only notice it in Quick Play where ELO is not affected instead of Ranked games.

Not sure that I've seen this addressed, but I view Elo as an approximation to a PageRank-like stationary distribution. Have others thought about this more formally?

Microsoft Research developed TrueSkill [1], where they frame the problem as a Bayesian inference one.

Incidentally, TrueSkill is implemented in Infer.NET, which IMHO doesn't get the attention it deserves. An open-source state-of-the-art factor graph engine with amazing performance. When dealing with many kinds of problems, including those involving discrete variables, I haven't found anything that compares in terms of speed and covergence.

[1] https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

There are a lot of similarities between classic Elo and Bayesian statistics. You essentially construct a prior for each comparison and update your prior by its deviation from the observation. Does true skill just do this in a more rigorous fashion according to Bayes' theorem?

> TrueSkill is patented,[3] and the name is trademarked,[4] so it is limited to Microsoft projects and commercial projects that obtain a license to use the algorithm.

Capitalism sometimes leads to stupid outcomes as this big issue with "what is property".

Imagine if physicists patented their equations, forcing universities, books, movies, wikipedia, etc.. all to pay for the privilege of using their equations. The world would still be in the dark ages with just a privileged social class with access to advanced knowledge.

Not to mention this will bar a totally valid shot of others human beings of getting into the same result while trying to solve this problem, giving they are using a math toolbox that is generally available to anyone.

Corporations should have the right to explore something they worked on, of course, but look at this example where Microsoft already implemented and still profit from their invention/discovery.

Using the legal system and not creativity to stop everyone else to progress using not just their knowledge, but the knowledge that can come from everywhere its pure stupidity as its not serving towards a collective good.

We are in the middle of the COVID outbreak right now, imagine if this sort of adversarial view were prominent in the world, how many more people would die if basic techniques about making vaccines were patented?

This is just utterly stupid and it's another proof that the system is broken.

We don't need to imagine:

https://www.doctorswithoutborders.org/what-we-do/news-storie...

> So far, pharmaceutical corporations and other manufacturers of products needed to combat COVID-19 have not shown any willingness to take a different approach during the pandemic to ensure the necessary broad access to needed products

> Furthermore, there has been an astonishing number of patents filed for COVID-19 vaccines in development, including more than 100 for the mRNA platform technology

I hope there someone with enough intelligence out there to understand that if there's a time to rethink the rules, the time is now.

Even for people that are only thinking in terms of profits and economy, if too many people die, there will be no economy at all..

People on top of social structures: government, corporations, etc, are working like pavlovian dogs unaware that the rules of the old normal doesn't apply to the new normality. Or its just a matter of a corrupt system that is organically formed over power and money that is completely oblivious about the values they should all stand for.

Probably could use something like this for looking at sports teams.

I've done this with various sports. In particular college football. The ratings were highly accurate compared to polls, except for undefeated teams not from major conferences. The polls involving humans would punish them terribly for not having the right pedigree.

I also rated the whole season simultaneously as it's my opinion that a ranking should be based on the performance in a season versus a power ranking of who is best now. By my calculations, the people choosing teams for the playoffs are either stupid or corrupt in the sense of not treating all teams fairly. It's a popularity contest.

As someone else has mentioned, the time-dependent nature is actually pretty important because PageRank (and there are a few others similar to this, Massey ranking is one...I can't remember them all) depends on every team playing each other team at some point and/or playing each other a certain number of times.

So, in practice, stuff like PageRank is fairly brittle, and ELO tends to work better.

I also don't think PageRank is approximation to ELO. PageRank are just eigenvectors so (I believe) rankings are proportionate to each other. This happens with ELO but to a lesser degree because you aren't necessarily looking at all the results for all other teams at all times. A lot of information is embedded into an ELO rating but you are updating match-by-match (I believe, I have done a lot of work with ELO and learned how to modify it so understand it well...I have far less experience with the matrix-based methods). So the practical advantage of ELO is actually a theoretical one, imo.

The best way to think about ELO is Bayesian updating: you start with a prior about team skills, you use this to create a forecast, and then update your ratings based on the actual result. Comparisons are Kalman filters, Markov Chains or MCMC, etc.

I will add ELO is very powerful. Yes, Glicko is better, ELO is a special case of Glicko but ELO is also far simpler than Glicko. Simple ELO models beat complex regression most of the time, it is remarkable.

Imo, it is the Bayesian-esque updating that works so well. And, if you understand what ELO is at this level, you can split this part of the model off and use it with regression or whatever you want. It is truly amazing though.

Super nit-picky comment, but it’s Elo rating, not ELO. It’s not an acronym, it’s named after the physicist Arpad Elo

Elo is essentially a time dependent graph algorithm. In a static scenario, Elo probably converges to something like pagerank, but I'm sure there are differences. It is worth mentioning that pagerank has been applied to sports in a similar manner as Elo.

Glicko2 is a better system.

https://en.wikipedia.org/wiki/Glicko_rating_system

Glicko2 has a fatal flaw for massively multiplayer games such as Pokemon Go. Basically, by intentionally losing certain games, you can increase the volatility in your rating, and then you can win a few games in a row against low ranked players and leapfrog people who are actually consistently trying to win.

See:

[1] Farming Volatility: How a major flaw in a well-known rating system takes over the GBL leaderboard. https://www.reddit.com/r/TheSilphRoad/comments/hwff2d/farmin...

This further strengthens my belief that Glicko 1 is much more preferable to Glicko 2. It's simpler (no volatility parameter), nearly as predictive (if not just as predictive), and it wasn't specifically designed for a monthly chess schedule, so it fits online games much more easily.

Depends on the environment really. If you're looking at measuring performance where competitors are not able to game the system (like, in professional sports) then Glicko 2 does a slightly better (but statistically significant) job than Glicko 1.

Does this flaw only apply to massively multiplayer games?

Depends what you mean by better. I agree Glicko approximates your true skill level better and faster. It's also not as fun as ELO in my view. With ELO when I win a few games in a row I get an exciting chance to play against much better players than usual. This is a fun challenge. It's necessary to learn new things. It gives you highs and lows. It's just fun.

With Glicko, once it pinpoints your true skill level, it feels like 2 meters of mud around you. You will never get out of it and you will be forever matched vs the same group of players stuck near you.

So yeah, I don't like Glicko. I have quitted services over it (even something like chess problem solving website). It's just not fun imo.

It sounds like it's not that you don't like Glicko but rather you'd prefer more variety in matchmaking. You can accomplish that easily by changing the way matchmaking picks your opponent.

There's also the TrueSkill rating system from Microsoft (developed for Halo), which claims to be better than Glicko:

"Glicko was developed as an extension of ELO and was thus naturally limited to two player matches which end in either win or loss. Glicko cannot update skill levels of players if they compete in multi-player events or even in teams. The logistic model would make it computationally expensive to deal with team and multi-player games. Moreover, chess is usually played in pre-set tournaments and thus matching the right opponents was not considered a relevant problem in Glicko. In contrast, the TrueSkill ranking system offers a way to measure the quality of a match between any set of players."

https://www.microsoft.com/en-us/research/project/trueskill-r...

For certain types of multiplayer games the recent Elo-MMR is even better (faster, provable guarantees) [1][2].

Quote from the paper regarding TrueSkill:

> The main disadvantage of TrueSkill is its complexity: originally developed by Microsoft for the popular Halo video game, TrueSkill performs approximate belief propagation on a factor graph, which is iterated until convergence. Aside from being less human-interpretable, this complexity means that, to our knowledge, there are no proofs of key properties such as runtime and incentive alignment. Even when these properties are discussed, no rigorous justification is provided. In addition, we are not aware of any work that extends TrueSkill to non-Gaussian performance models, which might be desirable to limit the influence of outlier performances.

[1] https://github.com/EbTech/Elo-MMR

[2] https://arxiv.org/abs/2101.00400

It is fascinating to see improvement in this space.

People -really- care about good ranking systems. I personally saw this when I administrated rankings for a local fighting game scene. Back then we used Glicko since matches were 1-on-1.

Trueskill is also under patent. Which is considerably more off putting than its rigor.

I noticed trueskill is pretty effective at matching people together, but the stickiness of rankings makes it frustrating if it’s user facing. People want to see the number next to their name change after 4 consecutive wins.

I think most people would want to see a significant boost if you won 40 out of 50 games. Winning 4 out of 4 doesn't mean much.

Is the issue multiple players or is it teams? I didn't have any problem implementing ELO for multiple players in games where there can be multiple losers but one winner. I modeled it as simultaneous wins for the winner, with a K-factor divided by the number of players.

But I wasn't concerned with being mathematically rigourus, maybe there is some glitch with this approach? It was for an EDH group. (BTW, don't do this. We no longer play EDH. It makes the spiky players even spikier, and then they get saltier when they're targeted as they have a far higher ELO)

I think the issue is that, outside of formal tournaments, there are rarely stable teams competing against each other in typical online multiplayer. each team is a random assortment of players, where anywhere from zero to a full team are queuing together. people might queue with a friend of a very different skill level. you need to track the skill level of each individual player and then estimate the win probability for a team of players who may never have played with each other before. I play a lot of csgo, and while far from perfect, I'm surprised how good the matchmaking is. I win pretty close to 50% of my matches.

> It was for an EDH group. (BTW, don't do this. We no longer play EDH. It makes the spiky players even spikier, and then they get saltier when they're targeted as they have a far higher ELO)

Yes please keep tournaments and rankings away from EDH for the love of god. CEDH is a different story because it's a completely different mindset, but if I sat down for normal EDH and was told we were tracking Elo I'd run far, far away.

It was mainly to track strength of different decks we all played, since you could check ELO of either. It was the strongest player who was upset by it, since once people realized he won 50% of 3 and 4 player games he was always targeted. But yeah, I wouldn't do it again.

First, a good article about TrueSkill I found when looking for systems to rank pairs of foosball players: http://www.moserware.com/2010/03/computing-your-skill.html

Second, apparently Tinder uses an ELO ranking system for their algorithm. I would add a link but everything I'm finding online right now is very clickbait-y.

I'd be interested to read more about Tinder's application. Off the top of my head, i don't see how it would apply to a dating app.

>What was it, though? It was a part of our algorithm that considered how others engaged with your profile. ... this part of our algorithm compared Likes and Nopes, and was utilized to show you potential matches who may be a fit for you, based on similarities in the way others would engage with profiles. Based on those profile ratings you received, there was a “score” — in the sense that it was represented with a numeric value in our systems so that it could factor into the other facets in our algorithm.

From: https://blog.gotinder.com/powering-tinder-r-the-method-behin...

Its not a deep description at all, but it sounds like when someone viewing your profile is treated as a game, which way they swipe is a win or loss for you and your change in score depends on their score as well? Basically if a desirable person swipes right on your average profile, its indicating you are desirable and moves you up more than say the inverse scenario where an average profile swipes right on a desirable profile (I guess this would be akin to a low ranked player losing to a high ranked player).

Edit: I guess the outcome would be people of equal (as perceived by others) attractiveness being matched.

> 3. Exponent Base (Seen as constant 10)

> In the articles I’ve read I have not ever seen this changed nor was it mentioned that a change could be used, but if you’re designing a special system and perhaps looking for something else to tweak this might be an option.

It can be easily shown that changing the base is equivalent to scaling the exponent, so if you're already scaling the exponent by a semi-arbitrary 1/400 chosen for aesthetic purposes, there really is nothing that is added by discussing the base of the exponent separately.

There was a quote from day[9] that really struck with me and wonder if anybody has some intelligent opinion on it.

"MMR [in context of systems like elo] is __not__ a measurement of skill, but of progress".

Depends on how quickly you calibrate to your 'true rating'. Elo-like systems take relatively quite long, since you can gain at most k points per game (but obviously fewer points per game if you're being matched with someone who's similar in skill to you). Chess GMs for example stream their 'new account to GM' runs, and it takes very many hours.

Chess is a game where there's very high correlation with individual skill and the outcome, in more noisy/random games like Dota 2 for example (which also are ~40 minutes on average) your expected relative gain per match is much lower.

Use it for class rank, replacing GPA. Being the best in a given classroom is a win, being at the median is a tie, being at the bottom is a loss.

This would help with the problem of people avoiding harder classes due to GPA fear.

Nice use for gaming

Crowd-behavior ranking systems should be banned for human relationships, where the person doing the ranking didn't intend to send a message to the crowd

How dependent is the result on the order in which players play? Assuming who beats whom is fixed, how much do results vary over the order of the tournament?

I believe order is important. The way to verify this is by running a simulation with players whose skills are all equal but vary with some normal distribution, and you will find that skill levels to do move about quite a bit even though the underlying skill level is fixed (I am thinking about order as just random shuffling here, so if we take a dataset where there is just randomness and in converges quickly then order doesn't matter...this doesn't really happen with ELO).

This isn't to say that rankings don't converge. They do. But things like order definitely can get in the way. Glicko controls for this by modelling uncertainty in a skill estimate (iirc, proportional to the number of matches seen). In practice, you would tend to burn the first few matches using ELO to try and get rankings to converge to a limit.

But, again, even then I have found that issues. ELO will quickly identify the worst and best but there is a lot of volatility around the middle grouping of teams. And this does vary with the underlying activity you are modelling too: activities with a lot of randomness take longer to converge. You can also control convergence with the K-factor.

I think this is one of the trade-offs that comes with the time dependent nature of ELO. You get more variance but less bias. Imo, it usually isn't a huge issue in practice because the upside of this is your method isn't static, and your ranking can respond to things like lineup changes (which tend to occur in many applications)...it works well in most applications.

Order is significant, but you can also technically batch the results for a tournament (or a stage of a tournament) to mitigate this: you assume that within your batch the underlying 'skill' of every player does not change - so you calculate the overall shift for each player but do not apply this total shift until you've calculated it for all matches.

Aren't the calculations done always with the initial points? I'm very curious to know if that's not the case.

In chess, at least, all calculations in a single tournament are indeed batched.

Reply on news.ycombinator.com