More on Dota 2
blog.openai.com
blog.openai.com
It's a great feat and kudos to the openai team, but it is VERY unfair for the human players who rely on a sensory interface vs a direct API connection. That's unlike chess or go where the interface isn't important. The really impressive feat will be an AI that uses the same sensory information to make decisions (and I really hope that's where the openai will head next)
Unless AI is constrained to pro player max pointer move delta, click rate, and vision latency, I don't really see much difference between AI and a team of kids running with aimbot shouting "cyka cyka".
I would guess that this already includes all the delays you are asking for.
gun.shoot()
as opposed to
image of a gun falls on the retina process the info sent by the eye synapses fired etc
:)
----
EDIT: On second thought, there might be a difference, in that this leaves the bot with more time to think, unless you limit the time of that. Not sure if that would dramatically influence the performance though.
but this doesn't undermine the achievement that they have made. It is phenomenal that an AI can play a complex game like DOTA.
Just like with network traffic, they are two different numbers.
Bandwidth limits are typically of the "You can only do 1 action (send 1 packet) 200ms since you sent the last packet", not "You can only send 300 packets per minute".
At the end of the day we can be nitpicky and speculate all day long, but we won't know for sure what they achieved/didn't achieve unless they publish something more concrete than a blog post that is intentionally written in accessible language, which has the side effect eroding some of the more specific measures that were taken to ensure a level playing field.
I.e., if an event occurs in the game, it's placed in a queue and the AI "sees" events pop out the other end of that queue, at a minimum latency and at a maximum rate (i.e. if too many things happen at once the AI is overloaded).
After that the AI makes decisions and puts the command in a command queue. The command queue works the same way: commands pop out the other side (to the game) after a minimum latency and at a maximum rate, to simulate the minimum roundtrip from input to action, and the maximum action rate.
I think OpenAI should show that the AI can derive (a close aproximation of) the API data from videos, but I don't think that building a closed training loop would add much value here.
Well they may be right about the "outperform" part but they are dead wrong about the waste of time/effort/energy part. I mean if (at least human-like) real-time video/audio recognition and decision making is not an impressive AI feat, I don't know what is. I'm no expert in the field but claiming that plugging into an API and crunching numbers is more important than sensory-based decision making, just doesn't sound right
You can pretty cleanly split that up into two different problems: "sensory-based data extraction" and "data-based decision making"
Though I haven't worked in the field of self-driving cars, I am fairly confident that they employ a similar split: One part that takes in all the (pre-processed) data from LIDAR, cameras, etc. and maps that to a simplified model of the surroundings, and another part that makes the driving decisions based on the simplified model.
Sensory->data mapping doesn't raise a lot of eyebrows anymore if you can generate as much sensory information as you want to explore all possible states, as it is possible with Dota.
Car AI is important because there are real life-or-death consequences, but the problem (again to my limited experience) seems more tractable: path choices are limited, action is rare, there's no team element and there's no competition. Even for human drivers driving a car in a city or motorway is a tedious, mostly repetitive task. Now, competitive driving raises the stakes and we haven't seen any self-driving car tackling that problem yet (which will definitely raise a lot of eyebrows)
I don't think that this is true, considering all the other humans on/near the road, that can influence the system by making actions of their own.
> There's too much action going on, too many visual/aural cues to keep track of, team cooperation/coordination, fog of war, etc.
I dont see how this is different from a car at all.
- too many visual/aural cues to keep track of -> everything you can see, car horns, etc.
- team cooperation/coordination -> as I said everyone else on/near the road
- fog of war -> blind spots & people hidden behind objects
From some experience in the AI world, I can say that I've seen systems that are good enough that they should be able to solve most of those subproblems in the limited system that is Dota. Yes, fog of war might also be interesting given that you have an agressive opponent. The real novel things lie at the strategy level, like picking a hero, buying items, etc., because, like AlphaGo, they demonstrate reasoning in systems with a large action space and delayed payoff.
The key problem is teaching the AI strategy and tactics. What heroes to pick? Where to lane them? When to rotate? What items to buy? What spells to level up? What enemies to target with which spells and in which order? These are the hard problems and they are very hard indeed. A 5v5 AI will have to become expert at risk calculation, Pareto optimization, basic military principles, and many more things. Compared to these problems, the choice of input mechanism is trivial.
I don't know why you think the visual/aural problem is trivial. The biggest achievement of AIs so far in this field is classification of static images
So, to me, based purely on the news coverage, it's not clear that it has learned anything like "superhuman" levels of strategy. We already know that computers have superhuman reaction times and precision calculation abilities, so it seems to me the interesting question is whether an advantage would remain after factoring those out.
This is partly it. From the interviews with the pros it mostly close to perfect micro, calculating results very well (it knows when it will win the 3 raze spam + 2 autoattacks with 2hp left and thus wins the match etc. for a human that would be hard to calculate) and due to those two it punishes every small mistake the human player makes very hard.
From the actual thinking parts it had to learn creep control (pushing too far just gives free exp/last hits to the other player etc.) and itemization (what counters what, when to get ward etc.)
I think there a little bit of strategy already in there since it learned about positioning, and not exposing itself too much (which is of course easier without fog of war). The only thing I was excited to see was the creep blocking, but the chance that this would turn out to be hand-trained was pretty high.
On top of that they also reduced the complexity of the game quite significantly by limiting items etc., which further reduced what humans could do against the bot. Even then, the bot utterly failed once humans were allowed to use a tiny bit of creativity.
And that's not even taking into account that this was not even close to the complexity of a real dota match. The big challenge in dota is in the decision making with incomplete information and in coordinating 5 people with only voice and the ability to ping the mini map, in a giant "search space" created by hundreds of different heroes, items and game mechanic interactions.
In terms of AI, I don't /think/ there's anything groundbreaking here. Correct me if I'm wrong, as I don't follow AI research, but this technology is nothing we haven't already seen. I believe the development of AI for Dota is a publicity move to get people excited about what AI could be for humanity. This might be the way to introduce AI to non-technologists and get people excited about it.
Yeah, I'm almost positive that this exhibition is intended mostly to raise awareness and create this hype. Go and Chess, for most people, are simple games compared to Dota2, so if Elon is worried about AI and want people to be more aware of the threat he perceives it certainly helps to make this big show and get all those impressions with a game that is considered by the majority of people (especially younger) to be more complex/harder than what has been done before.
Several years ago, League of Legends released an upgraded suite of bot characters. One of them, Cassiopeia, had to be turned down enormously from her best play to make her viable. She could beat many of the game's devs (mid-tier hobbyists) and was a non-zero threat to professional players. This was, to my knowledge, achieved with little or no machine learning at all.
The defining traits were similar to what we see here. She had area of effect spells with casting delays, meaning that the ability to precisely evaluate how other players could move was crucial. And she had a spell which refreshed based on the effect of those AOEs, meaning that millisecond precision was a major source of her ability to deal damage. And her ultimate was an exceedingly touchy and unpredictable disable based on the angle opponents were facing (in a game with instantaneous turning). Even top-tier pro players regularly lost its effect because of latency or judgement issues.
The results, by all accounts, were terrifying. She was barely competent strategically, but as long as she could afford items (and admittedly, the LoL bots don't need to farm) she could win all of her tactical fights simply by inhuman precision.
The OpenAI project is more admirable than that. It uses real farm, makes item purchasing decisions, and apparently has a rate-limited API. (That last seems especially important.) But I still wonder how much of the bot capabilities are derived simply from inhuman accuracy.
The bot would be more impressive if it used vision or buffer to analyze everything. The bot has all of the information which a human does not have. If the human had the same information I think it would be a better competition.
To me it seems that this bot is good only by the fact that is has information and input advantage.
I also want to clarify that this is not an unflavourful feat and that I think it is cool. But I noticed that a lot of members in the Dota community did not know of the bot API which is likely the case here.
Read the whole thing.
But still, it's learned "human" levels of strategy, and that actually amazing. And let's not forget, that this is only beginning, more stuff to come. IIRC - some third-party StarCraft AI actually invented a trick that was later used by human players on pro-level tournaments.
This certainly is an advantage for the bot. But it's also clear the bot understands some sophisticated parts of dota. It understands how to control the creep equilibrium via the aggro mechanics. Skill at this is one of the things that separates pros from casual dota players. In the video footage you can see sumail and rtz are surprised by this aspect of the bot. It also understands how to cancel salves as well as bait with them. That's all pretty impressive for a bot trained up from self play.
Not only do challenges come in the form of input mechanism but potentially many rewards as well in terms of how humans process and off-load information.
While I don't think DotA would be the most important example, I think having a more realistic interface is a good step.
[1] https://en.m.wikipedia.org/wiki/Embodied_cognition artificial intelligence section probably most relevant
hardcoded by a pro dota 2 player hired as a consultant
Impressive? Yes. Interesting? Not as much. We consider AI important not because it can play Dota. We consider stronger AI important for making our lives easier by solving problems. For AI to solve our problems we will not unnecessarily restrict it to our sensory information.
OpenAI could probably afford to do it, but it no smaller researchers or hobbyists would be able to compete.
Interface is one of the many.
If we look at a game such as WoW we can observe the fights in group content (dungeons and raids) have become more difficult over time. The WoW development team cites a few reasons for that: 1) players have become better so the difficulty has gone up, and 2) boss mods (software) have become better. This software aids the user in observing/notifying (sensing actions) , and executing (processing information and deciding upon it) mechanics on a fight and are available thanks to the LUA engine and API. Ironically, even without such software, that game has improved majorly in communicating mechanics to the player over the years (WoW is from 2004).
Hence aspects like the UI and API are going to affect the quality of gameplay of humans as well. For AI it would only be API.
Furthermore, there is network lag, interface lag, input lag, and cognitive lag. Only the latter seems fair game to me.
whether you agree protos was over-powered or not, I think we can all agree that game developers can benefit from AI with human limitations. it can help them design better games
But we shouldn't be down on this while we were going gaga over a lego sorter done the same way a month or so back.
It looks like for some things we can almost have a plug and play ai solution. EG, like we are seeing with image classifiers, this doesn't take years of phd doctoral research and game theory to build up a world class bot. Which is what everyone used to do. This is moving some of these techniques into the "get data set, get hardware, download library, train" plug and play type solution which we're seeing more and more with in other areas like machine classification. Eg stuff anyone with a few years of experience can do, maybe not amazingly, but better than they could hand coding the solution. The problem becomes one of gathering good training sets or building an accurate simulation to train in.
This means, I think, that you'll see way more of these types of ai solutions where people would have balked at a hand coded solution before. This in turn looks a lot like mobile's change to computing where things that were annoying to do on your home pc became different just because you had a camera + gps + computer + radio in your pocket.
I know my company has started using classifiers a lot more for things that are kinda sliding bad user actions instead of coding up huge rules engines. We may not be as effective as a several area deep engineers writing rules and doing data analysis, but instead we have 1 engineer per problem space being about 70% as effective which is still a huge win over not solving the problems at all.
The funny thing is that this bot actually pulled off the stereotypical hollywood training montage with just a few weeks of hard work it beat the best in the world. Just get some sweet rock in there and you've got it all.
I've heard this many times and conceptually I get the principle, but I have a hard time understanding how you create a legitimate starting position or measurement mechanism beyond "losing/winning".
(1) Not creepblocking at all, and letting your opponent have your creeps under their tower
(2) Modest creepblocking to punish (1)
(3) Severe creepblocking to punish (2)
Would be better than some hand-trained RL creepblocking which is divorced from game outcomes.
My understanding, from talking to a ML friend this morning, is that the latest progress is taking reinforcement learning and applying deep learning approaches (nets, etc.) to it. The key becomes finding the right scoring algorithms to tweak the neural net correctly towards the desired outcome.
The self-play really is the reinforcement side of things at work. How you take that 'score' and use it to correctly modify the input weightings - be them in a neural net, traditional algorithm, etc. - is the key.
Does this not become something similar to supervised learning if you are scoring internal states of the game? (i.e. scoring on more than just the outcome and things that violate the rules?)
I e, most robots fled and hid in the corners. I added additional critiera for hitting enemies, and the robots fled while randomly shooting bullets and hid in the corners...
No epic fights.
Hmm. Maybe I have the code somewhere...
And then selling the AI to Firaxis.
Yes, Civ 6 AI still isn't anywhere close to what it should be.
In particular, I'm stunned that the pro players accurately assessed "Sumail will win" on the 9th, but the improvements of one day of training invalidated the assessment.
(Assuming the original article didn't fix their title)
I don't know enough (no more than a layperson) about AI to have any meaningful comment there. Do they need to train the bot on every hero the same way or does it only need to relearn the hero specifics (and not items/strategies)?
Sad they had whitelisted item builds. I thought the whole point of a machine learnign bot was it was supposed to learn these themselves.
5v5 full game is way more complex than Starcraft. Hope OpenAI are ready.