Hacker News Reader

Comment by Benjammer | Hacker News Reader

Keep in mind that it's not just pure learning from zero, with these crazy emergent behaviors coming out. There is a fair amount of "coaching" involved. The OpenAI team explained this last year when they revealed the SF mid 1v1 version of their Dota bot.

People were astounded that the SF mid was creep blocking at an extremely high level, since wave positioning/management is a very complex behavior/process that doesn't pay off directly right away. The OpenAI team said that they basically taught the SF specifically how to creep block, and they've done this with other behaviors as well, like denying allied creeps, as well as a lot of the meta-level strategic play like warding/vision and item builds/usage.

View on HN

The thing is, OpenAI are terrible at the game. If they try to coach too much, they'll ruin their chances at beating the top Dota team. They have to let the bot discover its own winning strategies.

For example: There are an infinite number of places to place a ward. One way to train the bot is to preselect all possible ward locations, reducing it to 30 or so common ones. Another way is to make an optimization algorithm where the bot focuses on trying to maximize the "strategic vision" (if it's possible to come up with a measure for "strategic") and then let the bot place wards wherever it wants. After hundreds of years of self-play, it should figure out the best place and times.

As I write this out, I think you're probably right. There are too many aspects of the game for a purely-random algorithm to be effective... E.g. item builds. But I'm holding out hope that's just because they haven't figured out a good way to encode all dota items into a distance measure.

AI has proven over and over that humans aren't so special. And humans know how to adapt to the game.

That said, I wish OpenAI would be completely transparent as to what's emergent behavior and what's not. :)

Oh, one last interesting thing: Icefrog is going to roll out a big patch after this TI, just like he always does. I wonder how much of the bots' knowledge will transfer over? Or if they'd be better off training from a clean slate?

During the OpenAI stream where they first played against 5 other players, they said that the AI isn't picking items at all, it's just using pre-made recommended item lists. They also said that the match was being played on an older patch so they may just not update if the upcoming patch makes large changes.

>I wish OpenAI would be completely transparent as to what's emergent behavior and what's not.

Totally agree with you here. This would also help actual DOTA players to analyze what is truly emergent behavior that might open new doors to high level play styles, and what is just weird coaching/niche optimization choices by the OpenAI team.

For what is worth, they have been very clear about that kind of stuff in their blogposts and interviews. It just happens that the information is a bit disjointed due to that format.

>Oh, one last interesting thing: Icefrog is going to roll out a big patch after this TI, just like he always does. I wonder how much of the bots' knowledge will transfer over? Or if they'd be better off training from a clean slate?

The bots aren't even playing real Dota, let alone current patch.

That's not really true. There are restrictions, but it's real dota. The fact that people haven't had time to adapt to the rules is irrelevant; the bots' strategy is very similar to TI4, when deathball meta was in fashion and games would end in 13 minutes.

OpenAI are here to compete on equal footing. They aren't going to stick with the old patch, I would guess.

Watching the showmatch as a dota player, it was clear it's not "real" dota. I played Turbo while it was new, and the game they are playing is much more similar to that. Having _almost_ instant regen whenever you need it is a gamechanger.

It's sort of a "Theseus's paradox" situation.

Until the AI is thinking about how to deal with a possible 5th pick brood/huskar/meepo it's not real drafting. Until it's coping properly with courier snipes mid, courier use distribution etc. It's just playing a variant of dota.

It's still really fucking good. But it's not dota _yet_

re "That said, I wish OpenAI would be completely transparent as to what's emergent behavior and what's not. :)"

This is a pretty good summary - the main bit to know is reward shaping https://medium.com/@evanthebouncy/understanding-openai-five-...

> AI has proven over and over that humans aren't so special.

I disagree. I think AI has shows that we are very special.

Perhaps one day they’ll be able to create the analogue of what AlphaGo Zero (learning from scratch) was to the original AlphaGo (learning from human play). That would be very impressive.

Imo, going from OpenAI Five to some kind of OpenAI FiveZero or whatever is like 1000x the amount of progress of AlphaGo to AlphaGo Zero, just based on the complexity of game mechanics and the fact that it's 5v5 and not 1v1.

and perfect vs imperfect information - the policy network has to forecast enemy positions and deduce enemy goals. At least in Go you know full game state

Reply on news.ycombinator.com