Hacker News Reader

Comment by Invictus0 | Hacker News Reader

Is this evidence of a broader leap forward in machine learning, or are these advancements domain-specific? In other words, could these innovations be applied to other fields and applications?

View on HN

I haven't read the paper yet but "AlphaGo Zero only uses the black and white stones from the Go board as its input, whereas previous versions of AlphaGo included a small number of hand-engineered features."

This sound super generic, so yes.

I think the fact that it's no longer using Monte Carlo tree search is a huge step forward in the generalizability of the technique. But go is still

- a perfect information game

- with a relatively small input size (vs. arbitrary computer vision)

- cheap to simulate

- discrete action space

- deterministic

This isn't to take away from the magnitude of the achievement, but the nature of the problem itself makes the result less applicable to many tasks we might want to use RL for.

Math research shares those qualities, except small input size if you include the body of all the already-known theorems as an input. I don't know if we'll see much smarter proof assistants soon, but it doesn't seem absurd to me as a possible development.

Actually, just converting all the already-known theorems into a form that can be computationally verified (not just convince a skilled human) would be an interesting starting point. This would really help the metamath project, and perhaps make peer review of mathematical research papers easier:

http://us.metamath.org/mpeuni/mmset.html

Except it is still using MCTS (heavily).

It still uses MCTS as its search algorithm. It no longer uses random rollouts as part of the evaluation, though. (Previously it was rollouts/2 + value_network/2)

Random rollouts are what the MC in MCTS stands for.

Without that, it is simply a tree search.

Excerpt from the paper:

> [AlphaGo Zero] uses a simpler tree search that relies upon this single neural network to evaluate positions and sample moves, without performing any Monte-Carlo rollouts.

Does this mean it learns what to search? I wonder why they thought it was a good idea. I thought the whole point of MC was that pruning algorithms like the ones in chess wouldn't work for a larger search space.

That's what the policy is. Given a board state, the policy gives you a distribution over all available moves.

The policy network is a function from board states to a scoring of moves. The policy network with the greedy heuristic, ie pick the highest rated move with no explicit look ahead method, plays at a high amateur level.

This was... unexpectedly good.

It effectively reduces the branching factor of Go from the number of moves available, to the number of moves actually worth considering.

Are you sure the new version is not using MCTS ? I saw it mentioned in the new paper

It's further advancement of reinforcement learning which is notoriously difficult to train into usefulness.

Reply on news.ycombinator.com