Hacker News Reader

Comment by deegles | Hacker News Reader

Is there a way to learn Go from scratch using these AIs? I wonder if it would pay off in the long run to be fully trained by one.

View on HN

What you can do is checkout the algorithm at particular stages of development. AlphaZero&Friends start out not being very good at the game, then over time they learn and eventually become super human. You typically checkpoint the weights for the model at various stages. So early on, the algo would be like a 600 elo player for chess and then eventually get to superhuman elo levels. If you wanted to train using an AlphaX algo, you can gradually play against underdeveloped versions of the algo until you can beat them by loading up the weights at increasing stages of deveopment.

If you're curious how it would work, I implemented AlphaZero (but not Mu yet) using GBDTs instead of NNs here: https://github.com/cgreer/alpha-zero-boosted. Instead of saving the "weights" for a GBDT, you save the split points for the value/policy model trees, but the concept is the same.

thanks for sharing! this is very interesting. why did you use GBDTs instead of NNs?

> Thanks for sharing!

You're welcome.

> Why did you use GBDTs instead of NNs?

I mostly wanted to build an implementation to see how it worked; I was more familiar with GBDTs than NNs, so I figured I'd start with that. At its heart, AlphaZero is the marriage of two great ideas: using a Monte Carlo Tree Search (MCTS) to efficiently look ahead and find good moves and using a powerful ML model (like a ResNet) as a bot's intuition about which positions are good to be in (value network) and which moves are good when you're in which positions (policy network). So if a GBDT is powerful enough for your use case, the "ML Model" component in the MCTS+ML Model AlphaZero setup should be able to be swapped out with it if you want.

But I was also curious if GBDTs would do almost as well as a NN, because GBDTs can be much more efficient w.r.t. cost/energy. At the time when AlphaZero came out, I think it cost >$10M to train a superhuman Go algo. Nowadays KatoGo [1] can do it for <$50K. The most expensive part of training is the self play. You basically have bots play millions of games against each other and learn from the results of those games. Getting value/policy predictions each move from the ML models is a majority of the computation during self play, so if you make that more efficient, you should be able to train a bot faster/cheaper.

Check out this HN thread if you're interested in more AlphaX shenanigans: https://news.ycombinator.com/item?id=23599278

[1] https://github.com/lightvector/KataGo

will do, thanks for sharing this other HN thread. also sent an email to the address listed on your github profile.

You can play against open source reimplementations of some of the ideas behind AlphaGo family AIs. LeelaZero was one of the early ones, KataGo is probably your best bet right now. Sai is also in the mix.

All are _very_ strong. KataGo is ungodly strong, it beats pros.

Learning Go is about more than just playing against strong players, but it could help. The biggest difficulty is that the strong AIs aren't actually that good at playing handicap games, and they're also almost completely unable to explain to you why you should play one move over another.

> strong AIs aren't actually that good at playing handicap games

You wouldn't know that from watching JBXKataBot playing on KGS with a typical 5-7 stone handicap.

For the use being discussed (teaching beginners), you really need 9 stones, and it needs to work well. 7 stones is getting close though. I'll take a look at those games.

Last I saw I remember katago playing up to maybe 4 stones pretty well but the games being poorer quality over that.

For reference the best human player ~3800 elo. Alpha go zero is ~5200. A 400 elo difference means the better player will win 99/100 games if I remember correctly. Board game ai is definitely in a league of its own.

> All are _very_ strong.

That's the problem. To learn, we need an AI that can be just a little bit stronger than humans, but at the same time we need an AI that makes natural moves, not an AI that makes great moves 90% of the time and clear blunders 10% of the time.

And playing go, the AI should be able to give handicap stones and play reasonable teaching moves.

I think it would just smoke you from the outset. As far as I know it doesn't have a structured intelligence it can scale back - it would make the optimal move every time, destroying you like it destroyed top-tier players.

I tried learning Go a little while back but hit a wall. Was thinking about trying this more gamified option:

https://www.wolfeystudios.com/TheConquestOfGo.html

My rudimentary understanding of most reinforcement learning systems is that there is an "probability of optimality" associated with each action. Wouldn't there be a way to make the AI take the Nth optimal move or vary the degree of optimality with each move?

For these you are able to pick out worse moves than the optimal from them, but that's not actually the same thing as "play like a beginner, okay now play like an intermediate". These things are still openish questions and if nothing else there's a lot of room for improvement in tools to help you learn and review games.

You can make KataGo play moves that keep the score roughly even since it has a trained score head, e.g. kataJigo [1]. This will keep the game even to your level, a nice way to train.

[1] https://github.com/sanderland/katrain#ais

I'm not sure that's a good idea at all for training, though it is a really neat trick.

For training you really want your good moves to be rewarded and your bad moves pointed out, but if the AI just plays up or down to match what you do instead, there's no signal getting back to you on how you're doing.

It turns out that taking a superhuman player and making it play like a weak human is surprisingly tricky. It's not so hard to make a weak player - you just take suboptimal moves instead of the best moves. But often these suboptimal moves are bizarre. A weak human chess player will lose their pieces as they fall prey to forks and skewers and so on - tricks that are hard to see coming for new player. But they will rarely actively throw a piece away by moving it into danger. Even a novice human is pretty decent at looking one move ahead. But to a chess engine, or an MuZero agent, a move that loses the queen immediately and a move that leads to a sequence that loses the queen in five turns are basically equal. And so an artificially-weak MuZero agent, or an artificially-weak Stockfish agent will tend to make 'mistakes' that not even a weak human would make. This makes them a little difficult to learn from.

There does exist research on how to make a human-like weak player: https://arxiv.org/abs/2006.01855

The basic idea is to look at weak human games and try to predict when a mistake will be made. But I don't know if there's any approach that can do that without access to a corpus of human errors.

Yes, you can just `pip install katrain` followed by `python -m katrain` to get started. Personally I would recommend at least reading abut the rules first (unlike MuZero).

I think the strength or lack thereof of your opponent is actually much less important than the strength of the AI you use to review your games. After each game you should study the AI's advice and learn the moves it recommends.

Reply on news.ycombinator.com