DarkForest: Deep learning engine for playing Go from Facebook research

This is perhaps the best of a hundred scifi novels I've read in a few years. These novels will make you question the wisdom of sending traceable radio signals into space. Stay quiet, it's a dangerous Dark Forest out there.

Both do the cool thing of combining Monte Carlo Tree Search (which used to be state of the art, and is like really smart brute force tree search) with Deep Learning, and there is not much more to DarkForest than that so we can describe that first. The Deep Learning portions involves training a Deep policy Convolutional Neural Net, meaning a neural net that takes in a Go position (actually both take in Go-specific features about the position) and outputs the best moves. Datasets from human moves are used to get this, so its pretty easy. This policy neural net is used to guide the Monte Carlo Tree Search, which just means using the net's move predictions to play out a bunch of games into the future to evaluate the best move. In classic MCTS you play out to the end of the game and estimate the 'value' of a move by just ratio of wins it gets at the bottom of the tree, but the estimation of value is a more complicated for both systems. In DarkForest they 'Use PUCT and virtual loss. Remove win rate noise'. The move with the highest estimated value based on the tree search (guided by the neural net) is chosen as the next move.

AlphaGo does a bunch in addition to this which is hard to sum up, but basically: 1. They train a fast and a slow (but better) policy net, using the same dataset. Both of these are used to guide the tree search. The slow but better is used less often than the fast net, so a lot of games can be played out in the tree search. 2. The slow but better policy net is improved through reinforcement learning - making the system play itself and learn from that rather than just the human moves dataset. 3. Since there is a policy network that can choose moves, one can reasonably assume we can also get a 'value' network to evaluate how good a position is. This is also done, and the value network is used as part of an equation to compute the value of positions in the tree search (there is also a much simpler hard coded value equation, and the final value used as part of tree search is a weighted combination of these two). 4. All this is run in an absurdly, radically distributed manner with an insane amount of compute power - on the 'non distributed' version AlphaGo uses 40 search threads running on 48 CPUs, with 8 GPUs for neural net computations being done in parallel, and in the 'distributed version' it uses more than a thousand CPUs and close to 200 GPUs. I don't even know if that's up to date, those number are at least a few months old.

To sum up: DarkForest combines a policy network trained by supervised learning to guide Monte Carlo Tree Search for move selection. AlphaGo uses a small/fast policy network trained by supervised learning AND a slow but better policy network trained by both supervised and reinforcement learning to guide monte carlo tree search, and computes the value of final position with a combination of a hardcoded value equation and a 'value' network derived from the slow policy network. AlphaGo is absurdly, radically distribued (which helps a lot). Oh yeah, it should noted the amount of engineering required for AlphaGo involved dozens of people (two dozen are listed on the paper), whereas Darkforest seems to have been mainly developed by two people.

I would outline the theory here, but it's kind of a book spoiler. So be wary if you have interest in reading it eventually and want to know what I'm referring to now.

I'm curious if it's for Torch, or perhaps there's something more fundamental about why it's a good language for AI that I don't know yet.

Translation: Google beat us and there's no point keeping this private anymore. ;)