Hacker News Reader

Comment by omarforgotpwd | Hacker News Reader

I’m by no means an expert, but I’m fascinated by the idea that a neural net playing against itself can substantially outperform a supervised learning approach with a large training data set. I mean, gathering training data and making sure it’s labeled correctly and all that is a huge hassle so if you could eliminate that step or even reduce the amount or quality of training data required that should be a big win for AI, right? Especially if doing this not only makes things easier but also improves the performance of the model.

View on HN

It's very cool, but I think it also requires a very specific "adversarial" problem with a well defined notion of success (winning the game). If your machine learning task is something more nuanced and harder to define, e.g. identify word synonyms, I don't see how you can get around having a training data set.

I played with evolving algorithms playing some kind of robot war (in java) almost 20 years ago, and beside the fact that you had to be veeery patient since evolving a generation took 10 minutes at best, I realized that it's very hard to create a survival criteria that matches your intention.

I e, most robots fled and hid in the corners. I added additional critiera for hitting enemies, and the robots fled while randomly shooting bullets and hid in the corners...

No epic fights.

Hmm. Maybe I have the code somewhere...

Reinforcement learning isn't a new idea - I did a Berkeley-based edX course on it a few years ago now and it was not state-of-the-art to my knowledge. That had no deep aspect to it, we just generated a reinfrocement algorithm that utilised a good measure of performance (specifically, it was pacman and the score value is pretty good at that) and changed a few algorithm weighting variables at each iteration.

My understanding, from talking to a ML friend this morning, is that the latest progress is taking reinforcement learning and applying deep learning approaches (nets, etc.) to it. The key becomes finding the right scoring algorithms to tweak the neural net correctly towards the desired outcome.

The self-play really is the reinforcement side of things at work. How you take that 'score' and use it to correctly modify the input weightings - be them in a neural net, traditional algorithm, etc. - is the key.

Why wouldn't algorithm reach a local maxima when playing with itself, or even degrade over time by opening up to unknown attacks?

One typically keeps pools of trained networks to combat this.

it does. the bot can be defeated easily with tactics it didn't encounter while playing with itself.

> The key becomes finding the right scoring algorithms to tweak the neural net correctly towards the desired outcome.

Does this not become something similar to supervised learning if you are scoring internal states of the game? (i.e. scoring on more than just the outcome and things that violate the rules?)

I don't know what they're scoring on; they've not elaborated on that. However, when I first read into OpenAI it looked like many of their game links were to simply view the screen. To that end, it wouldn't surprise me if they're identifying things you'd otherwise visually see in the game and simply working off them. To use internal states not immediately visible to the player would be sort of disingenuous IMO.

Just a guess, but wouldn't this approach used in isolation result in a high potential for getting stuck at local optima? It seems to me that the success of this experiment is the result of interaction with players in different MMR brackets.

Yes, it can. It's fairly common to keep a pool of learners which aren't identical for exactly this reason - either playing against prior snapshots, forking training, or lightly randomizing some components of the learner.

Reply on news.ycombinator.com