OpenAI at the Dota 2 World Championships
openai.com
openai.com
I'm pretty interested in seeing their 5v5 results as well. It seems like that will have similar results as the bots can coordinate, but it's still a bit up in the air.
I'm really not sure how well a bot like this would do with 4 human teammates though. I would guess the bot would be a strong laner, but fairly weak overall due to it's inability to communicate, though if it's really good at learning how it's current teammates are playing as it goes, it may do alright.
I'm also pretty curious about the match limitations: no shrines (regen), no soul ring (mana regen at expense of some health), no raindrops (fixed amount of magic damage block)
The shrine is inactive for the first five minutes of a normal game, so it's a non-factor in most 1v1 matchups.
[1]: https://www.reddit.com/r/DotA2/comments/1ka4bp/q_why_was_sou...
I'm reminded of the games played in the movie 'Her'.
I've played a few dozen hours, and I understand most of them.
1) "smoke ganks" means that a team goes invisible, in order to surprise the opposing team, and kill as many as possible in a short time.
2) laner: one of the main 3 roles that one of the five players in a team can take (e.g. support, laner, etc).
3) and so on : "shrine", "regen", "soul ring", etc, are all very specific locations or objects or concepts (regen = regeneration).
DoTA is a really, really interesting game. Very strategic, great lore and graphics, and a huge fan base.
1) Team level, which dictates what the team as a whole is attempting to achieve
2) Mode level, which dictates what the character is trying to achieve individually
3) Action level, which dictates the specific responses being undertaken
Smoke ganks would be a case of the team level instructing the mode level to simultaneously group then seek a kill, which in turn would tell each of the action levels to move across the map, attack, etc.
For reference: https://developer.valvesoftware.com/wiki/Dota_Bot_Scripting#...
Like if the opponent changed where one unit was moving within a lag delta and a different decision was made, but also seeing a teammate bot execute a decision first would alter the inputs for the others.
A similar comparison is if a Counter-Strike or Quake bot would instantly snap to your head getting a headshot. Like, yeah, cool I guess, and it would beat 100% of human players 100% of the time, but it's not even remotely as impressive as Chess or Go. The razes in particular are "skill shots" that are sometimes hard for humans to estimate whereas for a robot it's just simple math.
I will say that the one impressive aspect of the bot was raze faking, but I think that it was one of those cases where it was "coached" (same as the creep blocking).
What makes you think that? In the interview, they explicitly disclaim this.
- In game 1, when the AI triggers his flask of health regen (which gets dispelled if you take hero damage) it zones Dendi out knowing he's going to come in for an attempt to dispel his regen. It does this with lots of non-committal, pre-emptive shadowrazes. It doesn't do this any other time. It's aggressively predicting what Dendi may favor based on its own cirucmstances.
- In game 2, Dendi said he was going to try letting one creep ahead and see if it would give him a lane advantage. [very technical DOTA mechanics] What this does is if the opponent creep blocks better than you, their creep wave strikes your overextending creep without taking any damage, under their tower, thus significantly weakening your creep wave, making the tug of war swing very heavily in your favor. You counter this by also letting one of your creeps out ahead as soon as you realize it. The AI that was perfectly blocking all the way till the creeps met up in G1, as soon as it saw the leading creep let one of its creeps forward. A perfect response that is highly unlikely to have been coached but determined as the best choice to, what I'm guessing is to return the game to a familiar state.
- The bot cornered Dendi between some trees and his tower, and then with 0 fakeouts/hesitation shadowrazed him. It was not just a matter of knowing the target was in range, but also accounting for the probability that he could dodge it. Not 100% certain about the mechanics here, but I believe: Shadowraze has a ~.2s cast animation. It can be canceled during the first .1s (fake out), but after that it is committed to, you will complete the animation, spend the mana, and cast the spell. If the cast crosses this window and the opponent is capable of moving the short distance needed to dodge in this time, it is theoretically not something the AI can guarantee on pure mechanical skill. So cornering is an approach advanced players use to reduce the probability and room for dodging a skillshot, and I found it taking instant advantage of that opportunity very impressive.
Besides the obvious mechanical precision of a bot once it's made a decision, the other inhuman factor here is what Dendi mentions, that it has zero hesitation to take advantage of even the smallest opening it has available. A pro human who is even instantly aware of an opportunity, no matter how theoretically certain they are that they should take it, hesitates at least a little bit to think "is there a trap? Am I not considering something?"
Hesitation is an incredibly human trait.
If this is accurate, one of the things I'm most impressed with is that the bot figured out creep-blocking. (I can't find a good GIF, but this is walking in a wiggly path in front of the first wave of neutrals on your side, delaying their progress and pushing the lane towards you, which is good for ~reasons.)
Creep blocking isn't all that hard in dexterity--I am a terrible dota player and I can more or less do it. And it's one of the most common pieces of dota knowledge; every pro player does it and since it's relatively easy compared to a lot of pro micro, everyone else rapidly learns they should.
But nevertheless--the bot had enough games that it could randomly jump in front of the wave enough times that it noticed a win rate improvement for that slight wave push, and begin to do it intentionally? (And then get good at it?) Damn.
One thing I don't really know about Q-learning and the typical nets used for it: I am guessing it is likely that internally to the bot's evaluation functions, there is some learned feature whose activation correlates well to the location of the wave equilibrium (since that's a feature that correlates well with winning!) At that point, is it likely that the bot can learn in smaller increments--that is, it knows that pushing equilibrium towards itself is good, and thus randomly creep blocking a little becomes reinforced (rather than having to notice the creep block's effect on game wins?)
I think there is a deep link between simulation and AGI. An AGI would need to be able to imagine how people and objects would act and react in any situation, which is the same as the ability to simulate the world, or to imagine. We might be able to create small simulations like Dota2, but the real world will be much harder.
We run through conversations, physical events / muscle memory, are constantly predicting the world around us - we often don't even remember moving through the environment on regular routes unless something unusual disturbs our running predictions.
It's super interesting to see some of that come through, sure in a more 'brute force' manner, but we randomly brute force bumped through the world as children too, until we pruned our selection trees.
But in the future coaching might indeed just be a human expert talking in natural language to teach it how to be better! https://www.youtube.com/watch?v=L_jGDV_5gPA&t=6
You might also pull aggro from their creep wave, which I think should also win you the game in most circumstances.
This highlights the importance of gradual adversarial training: if the opponent was perfect, there would simply be no way to win, and hence no signal on how to gradually improve to eventually find the optimal strategy.
Also shows a weakness in most training algorithms today that don't have, for example, the capability to watch their opponent closely and try to mimic it's actions in case he's better than you. Humans do this so quickly you can see Dendi already tried replicating the AI's strategy after 1 loss.
The idea is that first generations run around dying so the optimal response is to just stay put and let the opponent die. This now has basically 100% win rate so both of them do it, lowering it to 50%.
The other part is that there are random variations to the strategy it uses. This it experiments - they won't stay in base forever since that is giving you only 50% winrate (based on whose creeps would push tower first).
This random variation then means that the bot might go out, autoattack a creep wave and then come back to to base. Or anything similar to that.
This new strategy now has almost 100% win rate and the stay in base is no longer viable.
If it went out and died to a tower, it doesn't matter, it can just not use it and there will be period of base AFKing again, but it only takes one improvement (pushing a wave or hitting tower couple times) to make the base AFK obsolete and force new progress.
"A strange game. The only winning move is not to play."
- AI "Joshua" in "Wargames", 1983
I still can't quite tell if this behaviour was coached to the bot or it figured it out itself.
I was surprised to learn that he is the CTO of Open AI.
> If this is accurate, one of the things I'm most impressed with is that the bot figured out creep-blocking
Same. It is such long set of moves to get a perfect block. To have such a long move-set figured out as something advantageous is very high level of planning from the bot.
I was part of the Warcraft 3 beta. When it started all the races felt so unique. It's a bit foggy now but I think, for example, all the elves buildings - which appeared tree like - would automatically regenerate. Once they started balancing the game, all these special traits fell away. All the races were homogenized and despite still being different, all played much more similarly. Honestly, it ruined the game for me. I feel if I had just picked it up upon release I would have enjoyed it, but watching all these unique traits and play styles fade away made me realize what could have been.
Think going to a theme park. It's very pretty and attractive but underneath it all, the experience is always on rails without any deviation.
That's one way to deal with complexity. I hate those games, because the moment you become human, and try to do something intuitive but undefined and unwanted by the programmers, you get penalized.
Dota doesnt do that, although the recent patch feels like a shadow of such design: but nothing like league or blizzard
That's what I loved about Ultima Online. Characters were basically "Here are 100 possible skills to learn and 5 base stats. You get 700 points to distribute among your skills. Good luck"
My fav part was that doing things that require strength made you stronger, doing dexterous tasks increased dexterity etc.
While fun, the game was also an absolute nightmare for beginners. You really needed to find friends that knew what they were doing to help you out. This of course created a wonderful community.
I completely agree, but Dota is the very rare exception to that rule.
If one heroes ability is "too strong", most games would turn it down 20%, but dota instead perhaps lowers the hero's armor by 1, or lowers their sight range, or base strength, or something even less intuitive and tangentially related.
What you get is 100+ very unique heroes, that somehow manages to stay fairly balanced. It's quite an amazing game, if you can handle the timesink.
You have to consider how every single one of them interacts with all of the others and even less serious imbalances will be present in the games all the times for players that are unable to switch to playing anything else.
On top of that, you then have items which change how heroes can play and, more recently, talent trees which can significantly alter how they play also. Dota's balance is such that a hero can play both a carry role and a support role depending on how the player builds and plays them, and there's a number of examples like that which do get done at the pro level.
That's an incredibly diverse scope to balance.
Each composition doesn't have to be perfectly balanced against every other either and the system of picks and bans helps.
As far as I'm concerned Dota2 is probably the most well balanced competitive video game out there.
which is the wrong way to balance a competitive game
Games are mostly based on the skills of the players, so should games be balanced by assuming a human can do what a machine can? Of course not!
A bot can usually make computations way faster and more presice than humans.
Let's say a bot plays against another bot, both with different characters. One character might seem really overpowered compared to the other, but that might only be because the bots can react to certain events thousands of times faster than a human. Without that reaction time the tables might turn and the other character might be clearly stronger.
So how fast should we assume a human can do a given action? How precise should we assume the player is at fast calculations?
I do see a lot of things it can balance, but letting bots learn by themselves with no limitations will definitely not work out well.
Add latency. Limit its actions per minute including camera movements and maybe even inertial limitations of input methods.
And even if the bot is not a perfect imitation of a human it still might figure out surprising things that would take human players years of real time to figure out by chance since they can only spend so much time on playing and experimenting.
AI bots such as this one are usually limited to emit actions every N frames, so they can't possibly react faster even if they can solve the perception problem. There is nothing interesting for AI advancement in competing for the best reflex speed - it's all about strategy.
[0] https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn...
Some people dislike the MOBA nomenclature because it comes from Riot (maker of League of Legends; a clone of dota). In addition the term is pretty vague. Call of Duty, Starcraft, and DotA could all technically be “MOBAS”.
ARTS describes it a little better but I don’t think that term is great either.
Managing farming times and item spikes is part of the strategy. Of course it's not a true RTS, but I think that term is a bit better than MOBA; I'm also biased against Riot.
[0]: https://en.wikipedia.org/wiki/Multiplayer_online_battle_aren...
Not really.
There may be some limited use cases where this beats traditional sandboxing, wargaming or milsim. However, the primary distinction is that DOTA and the likes have fixed rule sets (albeit large and complex), and a near infinite number of simulations that can be run, wherein the actual "battle" will conform to those rule sets and play variability.
It may not seem like it, but we (military) only have a tiny slice of possible data from the past few hundred years, that would be appropriate to train a DNN on. Not only that, there are few wars that conform to previous war patterns. There are some constants, and we study those, but they don't really give you what you need for troop movements and tactics the same way as is done in DOTA.
As a side note: I think the most impressive aspect of this AI is that it was trained without watching any human games. In contrast, AlphaGo was bootstrapped from a bunch of existing human games, which I imagine was crucial to its training. Being able to learn how to fake out the human player without having seen this action before is quite impressive.
> AI techniques, which are good at learning what is "in the data"
but have not yet been shown to be capable of thinking outside the box
> AIs would be highly vulnerable
> exploitable quirks in the AI
> established lack of robustness of neural nets
> clever human players will be quick to exploit them
Due to this mystical "cleverness" attributed exclusively to humans, I'm not convinced a majority of humans will ever allow machines to be considered 'intelligent', no matter how far the technology advances. I'd love to be proven wrong, though.
As far I know, no A.I. developed has ever met this bar. It seems that the nuances, inconsistencies, and total freedom of language make it a totally different kind of problem to solve than a simple game-playing algorithm.
Or maybe I'm wrong and getting an A.I. to pass the Turing test is just another kind of game-playing algorithm with different parameters. At any rate, we're quite a way off.
Perhaps you'll require the Turing+++ test be conducted with audio rather than text. Then it will be video. Then a physical android. Then it will need to prove itself better than the human in an IQ test. Then it will need to pass various real world success tests, because IQ tests are too synthetic. But those won't be good enough either, so it will have to live 80+ years as a human in the real world without being discovered. But that's not enough, because maybe its life was too easy -- the real human experience is characterized by pain and suffering as well as joy, so we'll also need it to somehow prove its experience is more real. And on and on and on. It's always something else -- with every step forward, the success will be dismissed and the goalpost will move forward.
I should say this isn't intrinsically a bad thing, since it pushes us to advance the field of AI more and more. But IMO this is too often promoted as a pessimistic criticism of the AI field, rather than an encouraging push of optimism.
Why must this notion of binary "true intelligence" exist, anyway? Can't we see that intelligence is a spectrum (most likely with multiple dimensions of proficiency), and AI is continuously advancing upwards?
This is the $20000 long bet from 2002 between Kapor and Kurzweil:
> Ray Kurzweil maintains that a computer (i.e., a machine intelligence) will pass the Turing test by 2029. Mitchell Kapor believes this will not happen.
A human that can solve complex equations in a few seconds would probably be considered highly intelligent, but a machine can do the same in nanoseconds. Would you say a calculator is an intelligent AI?
This would include the ability to communicate with their human hirers and collaborators in natural language.
[1] We can also think of the human brain as a massive structure of relatively simple neurons with methods of communications among them that can be translated to mathematical parameters.
Decision making is how quake pros can beat aimbotters, and starts to become very important in 5v5 DotA, particularly at a higher level. You'd see it in terms of drafting, ganks, map vision/control, map positioning, sneaking rosh, and other unexpected meta strats. Considering the real-time computation requirements, this is almost certainly a much harder problem.
So while it's entirely possible bot-team would win just through insane mechanics snowballing the game (ganking 0ms reaction times would be fun), or through good-enough decision making + insane mechanics, they could also be outplayed.
wonder, against this type of AI, if it's even possible to land something like meat hook or sacred arrow?
on the note of good decision making, maybe even a failed hook could be enough to exploit the bot into death. for instance, hero X chases bot back toward tower. hook comes in predicting the bots path. bot would have to either walk into the hook or walk back into the hero chasing it.
That is indeed, by no means trivial. I do think it's pointless though, that's just another exercise for the sake of making it another exercise.
So first it's laining, which is showing in the game.
Then is the ganking basically 2-3 player working together.
Above that that's 5 men team fight.
Then you have the strategy planning in the band pick phase, and in game movement coordination.
There are other things like item choices, in game communication etc.
I guess bots never tilt...
This 1v1 match is definitely a proof of the strength and maturity of modern AI.
It will be extremely interesting to see if they can train a bot to achieve the above intelligence. If so, I guess the game is pretty much losing a lot of its appeal.
You might think not beating bots is OK. Not for me and many people have been involved in competitive scene. The moment machine beats human, a large part of the competition is gone. The essence is that you want to be the best. But if all you can do is to beat some other inferior opponents, what's the point?
> Secondly.. Chess & Go have been around for thousands of years and they still are played en masse daily.
This is irrelevant.
Kiting is maybe a notable exception, but it is also restricted to a relatively small group of unit matchups.
See for instance the following papers:
- https://arxiv.org/abs/1609.02993
- https://arxiv.org/abs/1702.08887
- https://arxiv.org/abs/1703.10069
- https://arxiv.org/abs/1705.08926
Full disclosure: I'm one of such researchers, and I have authored some of the papers on the topic.
You can lose the game before your summoner even hits the field.
https://www.reddit.com/r/leagueoflegends/comments/4n5og5/com...
(At a pro level, obviously. At the level I play at, not so much...)
In much the same ways top chess players have learned from engines, I have to think top Dota players will practice against and study the hell out of this bot if OpenAI makes it available.
Not everything is replicable by humans... for instance I noticed it constantly animation-canceling razes and only finishing it if it was going to hit; a human will definitely mess this up. But other things can definitely be used. It was positioning somewhat strangely, for example.
If we can have AI that beat Pro at 5v5 in Dota, it would very likely be an AGI.
[1] http://www.criticalhit.net/gaming/dota-2-vs-league-legends-u...
Chess computers crush the world's best humans, yet here we are in meatspace, still vying to win the human world chess championship.
Also, 99.99999% of players realize they'll never ever be close to being the best in the world, yet they still play the game because it is inherently fun.
Human Entertainment is one of the few areas where i don't see AI taking jobs anytime soon, because that removes a lot of the entertainment value.
One of the OpenAI guys mentioned that they could potentially use the same technique in real life applications like surgery. Surgery is not "just" run on a computer it has a physical component too. Is that really the next step or was he just throwing out a random example people could understand?
This comment section actually seems ok, but usually you'll see mostly neutral or negative comments.
More of the latter than the former. He was talking more in broad strokes about ai research in general then this specific project.
For me it makes as much sense as comparing someone sewing pieces of clothes together with someone pulling a zipper. Did people also have races against cars all the time? Is anyone getting excited over some plastic not changing texture when submerged in water for days or even years, while humans get elephant skin rather quickly? I do find all of that highly interesting, but it's more a morbid curiosity than being amazed.
> One of the OpenAI guys mentioned that they could potentially use the same technique in real life applications like surgery.
Oh yeah, it will all be for benefiting the elderly and the poor, I'm sure. At some point, ever around the corner. It won't benefit the people who need to have surgery in the first place because the so called civilized world can't even deal with warmongers and power mad cops, can't reign in sheer greed and sociopathy -- but nobody who has excuses today will still have them when faced with fully automatic enforcement systems.
Generally, at least allow for the possibility that someone who is not utterly fascinated by something you like might not be less curious and progressive than you, but the opposite, and that what you think is the bigger picture being a fraction of what they see. At least until you actually asked the people whose comments you don't like.
It would be annoying to ask the same question 10 different times in a thread, I think a top level comment is better. Also, I didn't see any made up explanation in the parent comment, it looks like a straightforward question to me.
> the so called civilized world can't even deal with warmongers and power mad cops, can't reign in sheer greed and sociopathy
Can't this be used to shut down literally any conversation? It's obviously not possible for any one innovation or idea to solve all these problems, so it's not really that useful to point out. Would you dismiss any story that doesn't say "war solved"?
> allow for the possibility that someone who is not utterly fascinated by something you like might not be less curious and progressive than you, but the opposite, and that what you think is the bigger picture being a fraction of what they see
Where are you getting this from? Nobody said this, who are you attacking? This comment is so absurdly negative, and it seems completely unprovoked.
Firstly, much like most things a lot of the press releases have been pretty disingenuous. I'm pretty salty about Elon Musks tweet -
> OpenAI first ever to defeat world's best players in competitive eSports. Vastly more complex than traditional board games like chess & Go.
1v1 SF mid is several orders of magnitude less complex than an actual game of dota and probably a lot less complex than chess & Go as well. More than that, 1v1 showcases haven't been a thing for quite a while now for good reasons, the last time they did a 1v1 showcase was at TI4 (in 2014). When they did these the format was best of 3 with the first two games being QoP VS Puck and then the decider would be SF VS SF. This was done because QoP VS Puck is a very mind gamey, strategic, match-up whereas SF VS SF heavily emphasizes mechanical skill where you can snowball off of last hits and denies, giving the match-up more of a sudden death feel. You'd expect a bot to be very good at the SF match-up but struggle with the QoP / Puck match-up where there is a lot more decision space.
To go back to what I said about 1v1 showcases not really being a thing anymore: Back in Dendi's day mid was actually a solo lane but these days in a real game the mid has to constantly be aware of support rotations. The bots super aggressive positioning would be heavily punished even in fairly low MMR pubs so I suspect the pros have deeply ingrained positioning rules that make them hesitate rather than match the bots stance.
All that said, the bot is really impressive and I can't wait for the 5v5 bot next year. It will be super interesting to see what strats it goes for and how much it emphasizes the bots inhuman reaction times / micro skill.
That's insane that he figured out how to beat it so quickly. I feel there are other ways to cheese it too. Like maybe survive until 6 -> rush shadow amulet -> smoke -> activate and walk into lane during fade time -> ult when the wave/bot is on top of you. I bet the bot has never seen invisibility and wouldn't know what to do
Blizzard's Starcraft did the same thing for their recently announced AI project where Starcraft 2 got an API for the AI developers to work with:
https://techcrunch.com/2017/08/09/blizzard-and-deepmind-turn...
> Bot scripting in Dota is done via lua scripting. This is done at the server level, so there's no need to do things like examine screen pixels or simulate mouse clicks; instead scripts can query the game state and issue orders directly to units. Scripts have full access to all the entity locations, cooldowns, mana values, etc that a player on that team would expect to. The API is restricted such that scripts can't cheat -- units in FoW can't be queried, commands can't be issued to units the script doesn't control, etc.
In the video, we see the bot "creep blocking." For those unfamiliar with dota, players can use the model of the unit they control to obstruct the movement of allied computer controlled units in order to gain a favorable position.
I suppose it's possible that over millions and millions of matches played against itself, the OpenAI bot "invented" this behavior for itself. But it seems more likely to me that the programmers "built that behavior in."
This kind of learning is similar to GANs, where the discriminator learns from real data and the generator learns from the discriminator.
Complexity and simulations of what you have not yet encountered is something humans can do with ease. This is not something a bot can do in a complex environment like DOTA.
I'll be the first to admit that I was wrong and that AI is truly a thing once I see a bot like this one beat a pro-team in a 5v5. Until then, meh...
Impressive nonetheless, like others said, knowing the reward function would be interesting.
Just the blocking of the lane creeps at the start is already super-human and gives the bot a huge advantage.
[1] like lower actions per minute, latency, occasionally missclicking and not being 100% certain about distances
It will be interesting for deep learning, though, where the ideas are a bit more abstract. Perhaps humans will be useful for a while longer.
I guess I missed it and the stream link is now just a general link to the TI stream, which confused me for a bit.
I played DotA a few dozen hours, mostly against the computer, to learn. Then I played only a few matches against people, but I generally didn't like them.
Why? Because there were expectations that I would know certain things, or that I would perform at a certain level, and the other players were, on average, a bunch of young kids with tons of time to play DotA and their own weird jargon and acronyms for me.
I would strongly prefer to play against bots only, and be able to adjust the difficulty, to make the game always challenging and interesting as I progress in skill level. It could also make the "competitor" in me happy by knowing what my (objective) ranking would be.