Stockfish Wins Chess.com Computer Championship
chess.com
chess.com
N Engine Rtng Pts Gm SB Ho Ko St Fi Ch Gi Bo An
1 Houdini 6.02 3184 17.0 25 187.25 ···· =01 === ==1= 111= 1=1= 1=1 =1==
2 Komodo 1959.00 3232 17.0 25 183.25 =10 ···· ==0 ==1= =1=1 11= =1=1 1=11
3 Stockfish 051117 3228 15.0 24 173.25 === ==1 ···· =1= ==== =11= =1= 1===
4 Fire 6.2 3112 13.5 25 152.25 ==0= ==0= =0= ···· ==1 ===1 =1= 1==1
5 Chiron 251017 3013 9.5 25 114.00 000= =0=0 ==== ==0 ···· === ==== ===
6 Ginkgo 2.01 3052 9.5 25 110.00 0=0= 00= =00= ===0 === ···· ==== 1==
7 Booot 6.2 3091 9.0 24 104.75 0=0 =0=0 =0= =0= ==== ==== ···· ===
8 Andscacs 0.921 3100 8.5 25 107.25 =0== 0=00 0=== 0==0 === 0== === ····
You can often trade variance for performance in competing strategies. The amount you're trading depends on your opponent's skill and strategy.
If an engine has a way to learn the skill of its opponent, that might be another way to determine the ideal level of aggressiveness too. Also if you're behind in a game, cranking variance is often your best shot.
Do any engines metagame yet? That could make things crazy.
So there is also no luck for any engine getting a favourable order of opponents or so.
Funnily enough, I feel there's a lesson for startup people in this. I have seen many people (myself included) talk themselves out of big, bold bets just because they are very good at thinking up reasons something should never work.
Komodo 17.5
Houdini 17
Stockfish 16.5
Stockfish can still in principle reach the final this way if Houdini and Komodo have a draw, because that would give
Komodo 19.5
Stockfish 18.5
Houdini 18.0
Very tough for Stockfish at the moment!
Edit: But now I'm not sure of the original comment that only the two top engines from this round will go to the final; in previous years there were more rounds before the final, but I don't understand how many of the rounds have been eliminated this year.
It's kind of down to the wire now: currently Komodo and Houdini lead Stockfish by exactly one point, while Komodo is playing Houdini in the final game for both of them. Stockfish has exactly one game remaining (against Fire).
In order to reach the final, Stockfish needs to win its game against Fire, and also Komodo and Houdini need not to draw. Then Stockfish would be tied with whichever of Komodo and Houdini loses the current game, and I guess there would be a tiebreaker prior to the final.
If Komodo and Houdini draw now, they're both guaranteed to face each other in the final. People in the TCEC chat are pointing out that if they were human players, they would face a strong incentive to deliberately play for a draw or to accept one if one were offered. But presumably the software has no idea of this!
Edit: Stockfish did just defeat Fire so the top three are
Houdini 18.5
Komodo 18.5
Stockfish 18.0
N Engine Rtng Pts Gm SB Ho Ko St Fi Ch Gi An Bo
1 Houdini 6.02 3184 18.0 27 219.00 ···· =01 ==== ==1= 111= 1=1= =1== 1=1=
2 Komodo 1959.00 3232 18.0 27 214.75 =10 ···· ==0= ==1= =1=1 11== 1=11 =1=1
3 Stockfish 051117 3228 17.0 27 215.75 ==== ==1= ···· =1= ==== =11= 1=== =1=1
4 Fire 6.2 3112 15.0 27 180.50 ==0= ==0= =0= ···· ==1= ===1 1==1 =1=1
It's been over four years since I put fishtest up, and in that time, there have been over 20,000 tests submitted. The really cool thing is that this distributed testing framework is only possible with an open source engine. So instead of being a disadvantage (everyone can read your ideas), it turns into an advantage!
Here is the announcement of fishtest on the talkchess forum: http://talkchess.com/forum/viewtopic.php?t=47885&highlight=s...
Initial discussion of the introduction of SPRT into fishtest, which led to a dramatic increase in our ability to measure improvements in self-play, in a statistically sound manner: https://groups.google.com/forum/?fromgroups=#!searchin/fishc...
SPRT background here: https://en.wikipedia.org/wiki/Sequential_probability_ratio_t...
Basically, we use a two-phase test to maximize testing resources. First a short time control test (15s/game), using more lenient SPRT termination criteria, then, a long time control (60s/game) test using more stringent criteria. That combined with setting the SPRT bounds to allow us to measure 2-3 ELO improvements has allowed the progress of Stockfish to be almost only improvements. Previously when developing an engine, you'd make 10 changes, and if you were lucky, 2 or 3 would be good enough to make up for the other bad or neutral ones.
If you look at the graphs on http://www.sp-cc.de/, you can see that it just keeps getting better, one small improvement at a time.
Don't know if it'll still work but it did a year ago and it's a starting point.
For serious play i'd say it is chess.com, playchess.com(Chessbase parent), ICC and only then Lichess.
In some UI aspects Lichess does feel nicer and lighter.
That's just the nature of the open-source model, the money is sorely lacking for promoting chess professionally.
Maybe some day the Patreon model will transfer to attracting top GMs to Lichess but for now they might check out Lichess and then leave immediately such as was the case with Wesley So earlier this year.
By comparison chess.com has the money to attract strong titled players to its online tournaments and even sponsor strong over the board tournaments such as the recent Isle of Man super Swiss.
serious: ICC, for the competition. casual: lichess, for the dramatically superior UI and value than any other site edit: the personal analytics alone make lichess superior to chess.com and other long time inferior sites like chesscube.
playchess and chess.com are relics, though chess.com survives on promos and articles and such.
You might be right that ICC is more serious but chess.com has been making great strides in providing nifty web and online apps.
chess.com sponsors online and regular OTB tourneys like the recent Isle of Man super Swiss, not sure what ICC does.
I feel that ICC is the one that is a relic from 1990s.
In all fairness I had not played on ICC in 10 years ever since they tried to blackmail me in ordering a membership lest my 10 years of tens of thousands of stored games on them would vanish. (I started on ICC when Sleator split/took the code from original ICS in mid 1990s).
I just checked ICC online presence and played a few games as a guest, there was 1 GM and one IM online.
Even lichess has more titled players than that.
My bias is that chess.com offered me a full membership as a lowly FM while ICC didn't offer anything but blackmail.
Again, I could be wrong, but the impression to me is that ICC is still sleeping on its laurels from late 1990s.
Another note is that all the other chess sites (lichess, chess.com, playchess) have a notably more international feel.
Mostly because Lichess is written in Scala[1], but the engine probably helps a bit too ;-)
Love the tagline, "lichess.org: the forever free, adless and open source chess server"
Komodo has the highest ELO ranking. Houdini leads the more important http://tcec.chessdom.com/live_mobile.php championship format.
Code and data are equivalent (for example, instead of an opening book that says “if you play white, open with pawn E4” one could have an evaluation function that says “after the first move of white, having a pawn on E4 is worth a million points”), so how did they enforce that rule?
Regarding the enforcement of this rule: As all chess engines that competed are pretty well known (and sometimes even open source), I'm pretty sure that they (rightfully) trust the programmers not to do anything against the rules.
AlphaGo, for example, has definitely played well-known human openings (and innovated on them) despite having no opening book.
The rule is against "explicit" hardcoded opening books, to force the engines to calculate the best move each time, to encourage variation and (I think) to allow the developers to focus on building stronger engines instead of managing huge opening books.
The encourage variation part didn't really succeed in this format, as they all tend to converge to the same openings
Engines have contributed substantially to modern opening books but they haven't supplanted the existing knowledge. Humans turned out to be wrong about many sharp lines (which were refuted by computer) and the computer can find really interesting ideas in many positions (which would be nearly impossible for a human to find) but the old human-approved Best Openings are still standing tall after the engine revolution.
This may have been true when engines were still only marginally stronger than humans, but I haven't seen any evidence that is currently true. A few years ago Nakamura + Rybka (previous best program) lost to Stockfish.
The strongest players are not GMs, as far as I know, and a very important part of those games is trying to force positions where the opponent's engine might make a slight mistake
Here is an interview with the world champion https://en.chessbase.com/post/better-than-an-engine-leonardo...
Anyway, I was just making a slightly facetious point that all our existing openings came about by a slow, semi-random process in which people tried moves, played out the games, and then looked at the results.
Usually "real evidence" that the human choices sometimes are better than the machine ones is that the engines after a long time agree with the human choices.
What is the statistical power of a 90-game round robin? Would this be a publishable result with p < 0.05 (or the new 0.005) against the null hypothesis that Stockfish and Houdini (2nd place) were of equal skill?
Love the subtle product placement :D
I think research needs to be done on algorithms that are designed from the start to be parallel, but I'm not smart enough to do that.
In chess, a DAMN GOOD heuristic is that Queens are worth 9 points, Rooks are worth 5, Bishops / Knights are worth 3, and Pawns are worth 1.
A few additional heuristics regarding "two bishops bonus", "passed pawn bonus", "Castled King bonus" and the like work really well in Chess.
----------
There's no equivalent to that in Go, so the best researchers could do was throw machine learning at it and hope for the best. EDIT: I'm well aware of Joseki (aka Openings) and Tesuji (Go "Tactics"). But Tesuji are not very easy to program.
The alternative is maybe to apply the Monte-Carlo Search Tree to Chess instead of Go. All games end in Chess (due to the 50-move draw rule). But Chess is incredibly tactical, and its far more important to exhaustively see all possibilities 3 or 4 steps ahead rather than try to figure out how the endgame's position might look like.
So Alpha-Beta pruning is likely a way better algorithm for Chess. A Machine Learning algorithm would have to figure out how to beat the current state of hand-programmed heuristics, which is not only incredibly efficient (due to bitboards and other optimizations), but also incredibly powerful.
That's only a big deal in Go circles, where Go computers have only been super-human for less than 2 years.
Chess AIs are well known to have been super-human for decades now. Chess AIs are able to build the Chess game database backwards (ie: Tablebases) AND forwards (Opening Databases).
There's very little human knowledge involved in a 6-move Chess Tablebase: https://en.wikipedia.org/wiki/Endgame_tablebase . Indeed, computer-generation of Tablebases have REFUTED human knowledge, and modern Tablebases are way stronger than the entirety of chess knowledge that existed before computer analysis.
Building Chess AIs "without human knowledge" has been going on for a long time now. The thing is, its very difficult to beat the SPEED of an assembly-language tuned count of bishop or rook moves.
IE: The Magic Multiply / Bitboard method:
https://chessprogramming.wikispaces.com/Magic+Bitboards
----------
The Chess AI game is played very differently from the Go AI game. In Go AI, "superhuman" programming techniques are still in its infancy.
In Chess AIs, the game is mostly settling down upon speed, speed, SPEED. In many cases, a weaker heuristic that sees +1 or +2 moves into the future (due to being a little bit faster) is far better than a stronger heuristic that takes a bit longer to calculate.
As such, a huge amount of Chess AI effort is in optimizing routines at the assembly level. Or perhaps data-structures that allow Opening Databases and Endgame Tablebases to be more efficiently utilized.
In contrast, a lot of Go AI effort is in representing search trees... but not so much in speed or optimization right now.
You can optimize the Neural Network so that maybe it runs faster (using NN specific hardware like maybe TensorCores or whatever), but I don't think anyone has any way to "generalize" a Neural Network into faster code.
Sadly the developer was pinched by Google to work in DeepMind. We suspect this was to help work on AlphaGo.
The author, Matthew Lai, was not part of the AlphaGo team.
Current chess algorithm to evaluate board position is already very fast. A deep learning version needs to be trained for a long time and might still not be as fast. Maybe one day when training can be done faster and cheaper, there will be more interest in deep learning chess engines.
PS: The estimated hardware cost to train AlphaGo Zero is around $25M (2000 TPU over 40 days, or 1700 GPU years).
Watching the game, it looks like if there was a human player instead of the Houdini computer, the human would be easily able to force the game into a draw instead of a loss.
Nonetheless, as an intermediate chess player, the games were fun to watch because of how different the play style computers can sometimes have versus humans.
Basically it was very close and Houdini may actually be slightly better, especially at the abstract heuristic stuff. Although Houdini creates good positions for itself, Stockfish excels at brute force and somehow always manages to turn things around as it transitions into the endgame.
https://github.com/mcostalba/Stockfish/tree/ONE_PLY/src
Is there no DB or machine learning inside?
No, SF doesn't use machine learning, it's entirely heuristic.
> If the engine is searching a position that is not in the tablebases (e.g. a position with 7 pieces)
and it speaks a lot about tablebases. What is it?
It's still a heuristic searcher when it's not using those though.
They seem to have focussed more on making the Fritz engine a good tool for human analysis rather than the strongest possible engine.
Not sure what the new stuff is like the last one I bought was Fritz 12 some time ago.