A recent chess controversy
chicagobooth.edu
chicagobooth.edu
It is like asking someone to pick a random number between 1 and 1 million and then saying, “oh my god, it must not actually be random… the chances of choosing the exact number 729,619 is 1 in a million! That is too rare to be random!”
-Feynman, from Six Easy Pieces
You could look at a bunch of other metrics to identify cheating: how many errors/perfect moves^ and whether that's within the usual range. How well were the opponents playing? Etc
If you consider that Nakamura might have been having a good day/week, was already stronger than his opponents, and some of them may have had bad games/days, you can change something from "extremely unlikely" to "about a dice roll"
^ according to stockfish
I'm all for having a Hikaru/Magnus discussion--one of my favourite topics--but this just doesn't make sense
Also psychological games fall neatly into the scenario you describe. I play better and you play worse because I got into your head, or sent the noisy people to be across the hall from you instead of from me, so I slept like a baby and you didn’t.
Match play at the World Championship (where the two players play each other repeatedly for many games) involves a ton of inter-game coaching and work as each player's team goes over what went well, what went wrong, and how the next game should be approached.
Round robin play in small fields also has a significant amount of preparation because the schedule is known in advance, so players will know whom they have to play the following morning and will prepare accordingly.
I'm not comfortable saying that Hikaru does exactly 0 preparation for 3-minute Chess.com blitz games, but it's probably pretty close to 0.
If you flipped a coin 100 times and all you got are heads you really should assume it didn't happen by chance.
By starting the sentence with if, you are selecting the occurrence to look at.
If you said I am about to look at the results of this coin toss that happened yesterday, if it is all heads then I am going to assume it was not random, then you are making the claim before you have seen the results. You can still be wrong, but the chances of you being wrong is the rarity of the event.
During WW2, allies tried to guess the number of German tanks by observing the serial numbers on captured tanks.
https://en.m.wikipedia.org/wiki/German_tank_problem If, say, the serial numbers are unique, and come in sequence, if the five first numbers you see are all less than 100, it's a far chance that there aren't produced 200 tanks. (Provided some assumptions, of course.)
The funfact is that you get different results if you follow the frequentist or the Bayesian approach.
(The frequentist/Bayesian estimates should converge as you collect more observations.)
No. That's not it. In this case, if you properly control for all the factors, it turns out that the odds of Nakamura having that kind of a win-streak (against low-rated opponents) was in fact high.
This is entirely wrong and missing basic high school mathematics for non-theater kids.
The original claim is not archived, if you can be bothered you can track it down and do the correct 'hot take'. You can't just grab the first statistical principal you think of even if everyone else on Hacker News does.
Article - "it violates the likelihood principle", this seems wrong and Nakamura seems right, but you'd have to look at the original claim.
They were finding patterns in a long biased list of numbers, probably.
"For safety, I always bring a bomb with me when I fly"
"Why?"
"Because the odds of two people bringing a bomb on the same plane are so low"
I don't quite understand this objection? If I won the lottery at odds of 10 million to 1, you'd say that was a very lucky purchase. But if it turned out I bought 10 million tickets, then that context would surely be important for interpreting what happened, even if the odds of that specific ticket winning would be unchanged?
What they are doing here is sampling the data after the fact, and obviously one needs to take a uniformly random sample of a dataset for any statistical analysis done on it to be representative.
They start with a prior (very low probability), I'm assuming they use the implied probabilities from the Elo differences, and then update that prior based on the wins. That's enough to find the posterior they're interested in, without needing to look outside the winning streak.
I think the problem lies in the antecedent. Given all chess tournaments played, how often would we observe such a winning streak on average? If the number of winning streaks is near the average, we have no indication of cheating. If it is considerably lower or higher, some people were cheating (when lower, than the opponents).
Then the question is, whether the numbers of winning streaks of one person are unusually high. If we would for example expect aprox. 10 winning streaks, but observe 100, we can conclude that aprox. 90 were cheating. The problem with this is that the more people cheat, the more likely we are to suspect an honest person of cheating as well.
Again, this would be different if the number of winning streaks for a particular person were unusually high.
I don't know many other notable cases of Nakamura accusing players of cheating. Many players dislike how Nakamura conducts himself on stream and how he interacts with the chess community and this leads to exaggeration. It's simply wrong to compare him to Kramnik, who has dedicated many hours over the last couple years to accusing players.
https://www.chess.com/news/view/hans-niemann-lawsuit-dismiss...
From his Wikipedia article:
``` In an online blitz tournament hosted by the Internet Chess Club in May 2015, American Grandmaster Hikaru Nakamura accused Supi of cheating (Supi had defeated Nakamura).[2] The tournament judges accepted Nakamura's accusation, reverted the match's result, and banned Supi from the tournament. Brazilian Grandmaster Rafael Leitão wrote in his personal website, "Accusing him of using an engine in this match is absurd. The match is full of tactical mistakes. Nakamura played extremely poorly and, honestly, wouldn't have survived long against any engine given his terrible opening.". ```
Some years later Nakamura lost 4-0 and again insinuated that GM Supi used an engine.
Despite all that, Nakamura still published a video calling him a "legend" for once beating Magnus in 18 moves
Or, more mundanely, bathroom breaks. https://www.nytimes.com/2024/10/16/sports/kirill-shevchenko-...
Your iPhone can reliably beat the best chess players in the world.
Before computers put an end to the practice, long games used to adjourn overnight. https://www.chess.com/terms/chess-adjournment
> During adjournments, players could count on the help of other strong masters, called seconds. These seconds would analyze the position and tell the player what they should play when the game resumed.
You use a chess engine to tell you the best move - you can run a chess engine on a modern phone that will easily best the world's top human chess players.
The simplest forms of this are things like: "play online, chess engine open in another window", "use your phone hiding in a bathroom cubicle" and "member of the audience follows your game with a chess engine and signals you somehow"
There are also rumoured to be very subtle ways of doing this - like playing unassisted for most of the game, but an engine providing 'flashes of genius' at one or two crucial moves of the game.
Major competitions have things like metal detectors and time-delay video feeds hoping to make cheating harder.
I'm sure it would be a downer that I cheated but it would do them a favor by saying: "look, you cannot stop it. Time for something new".
It's not really cheating in NASCAR, but rather, "it wasn't in the rulebook".
[1] Yes, the ribald jokes have already been made
The way people cheat online is by running a chess engine that analyzes the state of the board in their web browser/app and suggests moves and/or gives a +/- rating reflecting the balance of the game. Sometimes people run it on another device like their phone to evade detection, but the low-effort ways are a browser extension or background app that monitors the screen. The major online chess platforms are constantly/daily banning significant amounts of people trying to cheat in this way.
Chess.com and Lichess catch these cheaters using a variety of methods, some of which are kept secret to make it harder for cheaters to circumvent them. One obvious way is to automatically compare people's moves to the top few engine moves and look for correlations, which is quite effective for, say, catching people who are low-rated but pull out the engine to help them win games occasionally. It's not that good for top-level chess because a Magnus or Hikaru or basically anyone in the top few hundred players can bang out a series of extremely accurate moves in a critical spot - that's why they're top chess players, they're extremely good. Engine analysis can still catch high-level cheaters, but it often takes manual effort to isolate moves that even a world-champion-class human would not have come up with, and offers grounds for suspicion and further investigation rather than certainty.
For titled events and tournaments, Chess.com has what's effectively a custom browser (Proctor) that surveils players during their games, capturing their screen and recording the mics and cameras that Chess.com requires high-level players to make available to show their environment while they play. This is obviously extremely onerous for players, but there's often money on the line and players do not want to play against cheaters either so they largely put up with the inconvenience and privacy loss.
Despite all of the above, high-level online cheating still happens and some of it is likely not caught.
Edit: More information on Proctor here: https://www.chess.com/proctor
Interesting; I thought I'd read that even the very best players only average ~90% accuracy, whereas the best engines average 99.something%?
Edit: Even lower-level cheated games are rarely 100% accurate for the whole game, cheaters usually mix in some bad or natural moves knowing that the engine will let them win anyways. That's why analysis is usually on critical sections, if someone normally plays with a 900 rating but spikes to 100% accuracy every time there's a critical move where other options lose, that's a strong suggestion they're cheating. One of the skills of a strong GM is sniffing out situations like that and being able to calculate a line of 'only moves' under pressure, so it's not nearly as surprising when they pull it off.
To compute accuracy, you compare the moves which are made during the game with the best moves suggested by the engine. So, the engine will evaluate itself 100%, given its settings are the same during game and during evaluation.
You get 99.9something% when you evaluate one strong engine by using another strong engine (they're mostly aligned, but may disagree in small details), or when the engine configuration during the evaluation is different from the configuration used in a game (e.g. engine is given more time to think).
Cheating is as simple as having somebody feed you chess engine moves from a nearby laptop running stockfish.
"... But had I started cheating in a clever manner, I am convinced no one would notice. I would've just needed to cheat one or two times during the match, and I would not even need to be given moves, just the answer on which was way better. Or, here there is a possibility of winning and here you need to be more careful. That is all I would need in order to be almost invicible."
For example look at the position in this video [1] from a recent game on Chess.com between Hikaru Nakamura and Fabiano Caruana (the title of the video says Magnus vs Hikaru because the video covers 3 of Hikaru's games in the tournament).
I linked to a spot in the video a little before the part where one simple message could changed the game because the host is explaining what Hikaru is going to be trying to do. Briefly, trading pieces off is good for Hikaru, and that's what he starts to do.
You can see from the evaluation bar this Stockfish says he is slightly better.
Then he plays Bg5 which looks like an easy way for force a pair of bishops off, continuing the plan. But look at the evaluation bar! It quickly swings from 0.2 in favor of white to 1.7 in favor of black. But black can only realize that advantage by playing RxN, a move that Fabiano did not even consider. He went on to lose the game.
A prearranged signal from a confederate that meant "Hikaru just made a game changing blunder" would very likely have resulted in Fabiano seeing RxN. It's a move that many would spot if they were given the position as a puzzle and so knew there was a tactic somewhere.
From the footage I've seen of Kramnik, I think he does believe himself, and is just generally very "salty" about losing (as the kids say).
and then in the publication itself:
>>The likelihood principle [Edwards et al., 1963] is a fundamental concept in Bayesian statistics that states that the evidence from an experiment is contained in the likelihood function. It implies that the rules governing when data collection stops are irrelevant to data interpretation. It is entirely appropriate to collect data until a point has been proven or disproven, or until the data collector runs out of time, money, or patience
Surely there is a difference when you look at someone who played 46 games online in his life and scored 45.5 and when you look at someone who played 46000 games and scored 45.5/46 once.
The difference is that Kramnik wasn't "collecting the data" but looked at the whole Nakamura's playing history and found a streak.
Another example would be looking at coinflips and discarding everything before and after you encounter 10 heads in a row to claim you have solid evidence that the coin is biased.
They are misapplying the principle here. If what they wrote was correct then someone claiming: "Look, Nakamure won 100 out of 100 if you just look at games 3, 17, 21, 117...." would be proving Nakamura cheated if they applied methodology from the paper even assuming one in 10000 guilty players. Just because you can choose sampling strategy and stopping rules (what the likelyhood principle states) doesn't mean you can discard data you collected or cherry pick parts that support your hypothesis.
How the data is collected is absolutely relevant and Nakamura is right to point it out.
ELO is presumably more accurate for over the board games at tournaments where players bring their A game than low stakes online games where someone may be less engaged. That’s IMO more worth testing.
The sound conclusion is that this is not evidence of cheating, but it is not evidence of the contrary either.
But, of course he doesn't. He streams all his games and gives constant stream of consciousness commentary. If you can explain your top level moves live with seconds per move, you aren't cheating.
In Bayesian analysis, probability does not refer to the long-term frequency but instead to the subjective credence given to the event. Otherwise the probability of any one-off event would be undefinable. Therefore it follows that you need to have a prior over possible hypothesis in order to update your beliefs systematically according to the laws of probability theory. If it were known that Hikaru had cheated in the past, but typically does not, we might use a different prior (e.g. a Laplacian prior in this case); if we knew cheating to be dependent on some other measurable variable (e.g. the emotional state of the player), we would incorporate this into our evidence.
The actual chess community's takeaway from this (if consensus is important to you) is that Kramnik (the accuser) has lost it a bit.
While ELO ratings are a probabilistic model, who said wins and losses have to be randomly distributed, there can be bad days and good days, for example if you haven't slept or if you are at the peak combination of study and cognitive, say because you are well rested on a monday and have been studying on the weekend.
What are the odds that a cheating accusation accurately identifies an instance of cheating?
I don't say this lightly: Kramnik very likely has some sort of untreated psychiatric disorder. He is effectively a lolcow in the chess community because he regularly (as in, almost daily) accuses much better chess players of cheating.
It's honestly a bit undignified to treat his accusation against Nakamura as anything other than a man yelling at the sky.
Sometimes you have to treat any accusation as “real” just to keep the cheaters at bay. (Cheating at online bridge is rampant, and cheating at bridge competitions was and perhaps still is fraught with many scandals).
It’s often common that the cheaters really ARE very good players - they’re just looking for less work, not a goal they couldn’t obtain otherwise.
why was it changed? This isn't a 'recent' story, it's from January.