AlphaGo beats Ke Jie again to wrap up three-part match

'data efficiency' -- that's a good candidate for the next buzz imo. People talk a lot about how massive amounts of data regressed over giant networks of dumb functions happens to perform well on many tests, but they ignore the fact that the performance of the system is sick. It fails if you tweak the input by small amounts that would never confuse a human. You shouldn't NEED that much data. You need better priors. Current ML systems rely on arbitrary coincidences to succeed at multiple choice tests. I did that in courses I didn't go to class. They are primitive and childish and won't scale with pure data/processing. We are still missing something.

There are a lot of problems for which you effectively have limitless data.

For example, optimizing the shape of an aerofoil, wing, wind turbine, ship, pump, etc. The "real" data is the result of computationally costly fluid dynamics simulations. The network uses as much of those as it likes to modify the design to be better.

There are also a lot of problems where there is nearly limitless data (ie. > 10^12 bytes of data). Examples would be MRI/CT scans, all internet video, all TV channels, etc.

You can view there being 5 dimensions/concerns with large scaling factors containing issues needing solving to be generally applicable.

1. How to search 2. How to resolve breadth concerns 3. How to resolve depth concerns 4. How to resolve large training requirements 5. How to form and resolve recursive goals

With AlphaGo, it looks like they may have solved 3

1. Search. Monte Carlo will likely be used from here on out. Much more efficient than full brute force search (chess style). 2. Breadth - Policy network. Reduced branching factor from 250 to between 5 and 10. 3. Depth - Value network reduced depth from roughly 75 to < 20

Next they're working on applying reinforcement learning to their two NNs. Ie seeing if they can optimize how well they can do and how quickly they can learn. Maybe there are large gains there.

If so, the final task would be finding a game or situation that required nesting of sub-goals like everyday life does.

They've made some impressive gains.

> In reality, unless you have a really good simulator, deep reinforcement learning is almost never applicable.

I could see Deep Mind applying this to something like catching a ball in the near future. Just to show it's generality. It's not that different of a problem space than learning to play breakout.

DeepMind claims that AlphaGo has already paid for itself when it was used to cut Google’s data center cooling costs, so it is not infeasible there may be many other domains which may benefit.

> AlphaGo essentially baked good movies into value and policy network by playing millions of times.

I don't think that's a very good description of how AlphaGo was trained at all; you're essentially saying it merely overfits the training set, yet it clearly generalizes rather well to unseen board situations and still evaluates them sucessfully. No machine learning system would be found usefull if all it could do is merely memorize the training data.

Re the use of deep reinforcement learning, well for one the role of reinforcement learning in the first version of AlphaGo, the one described in the Nature paper was rather limited, and a small part of its training; it just made a ~3d KGS policy network into a ~5d KGS bot, and used to generate a training sample for the value net. If we had enough recorded human games to train the value net directly, that'd be an unnecessary step anyhow. And you could create such a training set w/o reinforcement learning since there are pure monte carlo bots stronger than 5d KGS - but that'd be far more computationally expensive.

But its still not really true that there aren't obvious applications of deep reinforcement learning - indeed robotics is one promising application, and that seems rather relevant. this paper initially demonstrated an impressive improvement in manipulative tasks, and you can prob follow its numerous citations for newer stuff: http://arxiv.org/abs/1504.00702

I do agree that this exact architecture in AlphaGo prob doesn't have applications beyond just teaching us how to play go better; it seems too specialized. I believe they mean it in just the vaguest possible sense; that the kind of deep algorithms demonstrating incredible performance in AlphaGo have diverse applications; but this should not come as a surprise to anyone even loosely following what people have done with deep learning in the past couple of years anyhow.

Deepmind published a summary video for the game here: https://www.youtube.com/watch?v=5fOmbqjH7zI

What sorts of modifications could even be made to Go that it doesn't turn into an imbalanced mess?

It's not general AI, no one said it was. It's similarly not going to play checkers at the drop of a hat like KJ could.

The thought experiment a little more refined, like alphago programmers get time to implement only validMoves() and makeMove() then let the same learning process go on it, I wouldn't count it out. If you only flip the switch once the new game variant starts, before a human has even begun to take an action the computer will have played out thousands if not millions of games using MCTS and slowly training up its networks in real time instead of using a static one. As the game progresses the bot should only get better and better.

Two approaches used by the previous version of AlphaGo (MCTS and a NN for position evaluation) have been tried in the past in computer chess with relatively poor results, while in the past few years they were already giving good results for Go.

The current version might be using some different techniques that could be more useful, though. They should release some details in the next few weeks.

In both games, AIs find moves by way of a search tree that looks at the possible moves and looks for one leading to a win (see e.g. https://en.wikipedia.org/wiki/Minimax ). Since testing all possible moves (i.e. an exhaustive search) is infeasible, to be practical they instead use heuristics to decide the value of a given state of the game, so they don't have to search all the way to a win or loss every time. So the core of this is a good heuristic function to evaluate board states to guide the search (although in addition, a lot of the work that was done on chess AIs previously was about pruning the search tree so bad lines of search could be abandoned more quickly). I'm not sure what the state of the art is for this in chess, but one of the innovations of AlphaGo was to train a deep convolution network with data from millions of games with their board states and those games' eventual outcomes and thus creating an evaluation function (the 'goodness' of a state is its likelihood to lead to an eventual win). This may or may not be able to also improve chess AIs, depending on how close to 'perfect' the current best-of-breed heuristic is - someone else would have to chime in on this.

The other big improvement from AlphaGo was using another deep convolution network trained on moves from all those games to create an 'expand' function. Unlike chess, the branching factor of that search tree in Go is ridiculously high, because you can place a stone pretty much anywhere on a big board, while in chess the set of legal moves at any given time is fairly limited. So that neural network they use as an expand function allows them to prioritize the order in which moves are considered. This innovation isn't necessary for chess as the branching factor is much lower - but it might still provide an input for the prioritizing the order in which moves are evaluated.

After learning how all this worked, I understand why Deepmind themselves seem more impressed with their Atari AI, as it actually learned how to play the game from first principles by just looking at the pixels and being told its score, whereas, the approach above, impressive as it is, still actually just plugs the neural networks into existing algorithms, rather than use them for the core decision making process too.

There is DeepChess, which is a proof of concept that it can be done: https://www.cs.tau.ac.il/~wolf/papers/deepchess.pdf

I don't think it will necessarily be as good at chess, because chess has a much smaller search space. So "stupider" methods can actually work, while Go requires very good pattern recognition to prune the search space.

If you win two games of a three game match, the last game doesn't matter.

The problem with professional dan ranks is that they're capped at 9p, and each association has different rules for awarding that rank. Because of this there is a lot of variance among 9p.

For instance, Fernando Aguilar 6d (aguilar1 and aguilar on KGS), an amateur dan from Argentina, has defeated Hasegawa Sunao 9p as well as Yo Kagen 9p. In theory this should not have happened.

If the cap didn't exist, I think Alpha Go may be around 13p, based on the fact that Lee Sedol and Ke Jie may be able to give reverse komi to many top 9p players, and that Alpha Go may be about 3 stones stronger than both. Park Yeong-hun is less often mentioned but he may be playing at a higher level to that of Lee Sedol.

From what I've heard AlphaGo does maintain an estimate of what its odds of winning are at any given point in the game. Not sure if anyone from DeepMind has released any of that data from game 2 though.

As in AI taking over the world? No, since AlphaGo is optimized to do just one thing.

'Humans' are fucked one way or another. We were always fucked. Our parents and grandparents were always fucked. Did you expect to live forever? The future will contain our children, and they will be better in each and every way. And they won't be human.