DeepMind – StarCraft II Demonstration
news.blizzard.com
news.blizzard.com
Your AI would then be able to participate in tournaments and fight other AIs on the existing ladders.
Additionally, one could watch BW bots fighting each other on: https://www.twitch.tv/sscait
Well, that's fun. I mean, if it works, it works.
I know they have said they will be limiting the maximum APM of the bot to around 180 for the final bot but they may not have implemented it at the time this was happening.
Edit: another example https://www.youtube.com/watch?v=IKVFZ28ybQs
As far as I can tell, the AI must be cheating, at the very least to obtain map vision beyond what is available in a match. In order for the zerg AI to observe each tank's turret rotation and predict which zergling it's about to fire on, zerglings have a sight radius of 8, smaller than the tank's 11. In a fair match, the zerglings would be fired on before getting a chance to see the tanks. Even as the group of zerglings gets close enough to see the first tanks, other tanks start firing from beyond the zerglings' vision, so I'm not convinced this AI could pull this off in an actual match.
I'm not sure there would need to be that large a skill difference, relatively speaking - worker micro has a really high skill ceiling, and players start with 12 workers instead of 6 nowadays, so 2 extra is a much smaller difference.
Evenly matched armies looking for a micro-based edge turns basically into a fighting game like Street Fighter, where it's about reading your opponent, baiting them into a mistake, pre-empting them, taking decisive action, crisp reflexes/timing, doing something unexpected or unorthodox to get an advantage at a huge risk, etc.
Especially in ZvZ where spine crawlers can be built, it's not an auto loss -- it's a bet that the opponent is playing a little greedy, and a chaotic fight.
That's not a worker rush, that's an early pool where you brought some drones.
..what? Thats an absurd consequence -- sc/2 has a hundred different ways to rush, and they all come with the note: if it works, you win, if it doesn't, you're dead. And everyone accepts it. What's so special about drone rushing that it'd receive such consequences?
I mean hell, there's Has specializing in cannon rushing and goes damned far; but drone rushing? No, thats just too much!
It meant that the AI actually found a working strategy against one of the standard bots, where it had not found any before.
It will probably never work when it starts to play against itself, but we don't know if the AI is already at that point.
If it is a viable strategy and not a real edge case to punish some maximal greedy strategy, the game should probably be patched.
The problem is that there's a sort of rock-paper-scissors game of (conservative)-(greedy)-(all in). That is, a player who build defenses early on will crush a worker rush but they'll be far behind a player who ignores defenses and focuses on growing their economy. The worker rush is designed to punish this greedy build order.
There's nothing that can be done, really. If you decide to send a worker to scout super early on then your economy suffers so much that you'll be behind no matter what you do.
Edit: to those of you drawing a distinction between an immediate worker rush and something like an early pool or a cannon rush, that's splitting hairs. These are all classified as all-in strategies that almost automatically lose if they encounter a conservative build order.
The term means you take your initial workers and charge the enemy. If you know where the enemy is you'll arrive about the time they are up one or two workers on you, then you just have to fight when you're down one or two workers and you can't back off (they'll return to mine).
A human can win like this if they are so much better at micro that they can overcome the material disadvantage (unlikely) or if their opponent panicks and does something dumb (e.g. pulling only half their workers to fight).
Because worker rushes arrive so early it doesn't matter what plan you have for the match, you'll have at least as many workers (if you are also worker rushing) or you'll have more. Maybe if you're going six pool you'll have fewer but you'll also be able to get lings and win with them.
The computer can lose to worker rushing only because it can get caught in loops of poor decisions (e.g. you lure away a couple workers and kill them).
Worker rushing is always a bad plan.
This is common technique, mostly used to get scouting drone back through a walloff (own unit stands on "hold" in a small choke) or when a mineral line is under siege (e.g. Siege Tank, Liberator), the player can select all his workers to mine a single mineral patch (has to be spammed, otherwise workers redistribute) outside of the siege range until they siege is cleared up.
As the grandparent stated, a worker rush only works against the built in starcraft computer player, because the computer AI is too dumb to recognize "I am being worker rushed" and may not commit all of its workers to defending.
Things like early pools and cannon rushes are (depending on the details of the meta at the time) a rock papers scissors game. Cheese (early pool/cannon rush) beats aggressive play beats conservative play beats cheese (roughly speaking).
Worker rushes are just bad. Apart from playing against 6 pools on really old versions of SC2 on maps with ridiculously short rush distances they just lose. Even then I don't think I'd call them good.
That might solve its task, but it’s a far cry from learning to play Starcraft well. I was surprised they seemed to have trained it against the standard Ai as that’s not useful.
If it's playing against the Starcraft 2 blizzard AI, it will learn a lot of "bad habits" because outright bad strategies work against the AI. Thus the worker rushing.
Your "reasoning" is actually a false perception. It doesn't make these mistakes and tries strategies that seem improbable to human players.
A 50% win rate against the Insane AI is very good, as at the higher AI levels the computer actually cheats (no fog of war, more resources, etc).
The AI cheats, but is also incredibly predictible, and bad at decision-making.
If you do this long enough you can build a virtually undefeatable siege tank army that the AI would simply smash its face into, and then corral them to 2 or 3 bases. They would always run out of minerals while I expanded everywhere and slowly inched tanks forward. Doing this same strategy against humans got me to about gold tier in the early days of sc2.
It's not my fault if you don't scout. Planetary Fortress rushes, proxy barracks, cannon rush... it's all fair game.
> Scholar's Mate has sometimes also been given other names in English, such as Schoolboy's Mate (which in modern English perhaps better connotes the sense of 'novice' intended by the word Scholar's) and Blitzkrieg (German for "lightning war"), meaning a quick and short engagement (Kidder 1960).
> After feeding the agent replays from real players, it started to execute standard macro-focused strategies, as well as defend against aggressive tactics such as cannon rushes.
The reason is that while the attacking player sends over their workers to attack, the defending player is still constantly making workers. By the time that attackers get there the defending player should have more workers. He simply has to command all of them to fight the opponent's workers.
If the opponent doesn't react, you win because you kill all their workers.
If the opponent overreacts, you win because you are mining more than them (IIRC this is a problem with the insane ai).
If the opponent reacts poorly in other ways, you might be able to win by out microing them. E.g. SCVs (terran workers) consistently beat drones/probe (the other races workers) 1-1 without micro. Repairing can be an advantage. The offensive AI being better at stacking damage on a few units. Etc.
The defensive AI should learn to fix these problems and stop losing to worker rushes, but it could take awhile. In other words the defensive AI starts out as a lower ranked player.
PS. I'm not sure how serious a tournament this was, but here's an example of a real worker rush (not an early pool with workers attached): https://www.youtube.com/watch?v=_AtqgaJzJP4
Of course you lose all of that time it takes for your workers to drive across the map, so it's a pretty dicey strategy. A 50% win rate against the AI sounds about right.
Or is "understanding" an attribute which is only afforded to humans who do these things, much like a machine can process the same inputs in the same ways as a human but will probably never be considered to have 'qualia'?
Using this strategy is objectively bad in that it can be repelled by an opponent almost regardless of their initial strategy.
If deepmind’s goal is to beat the standard Ai, then sure I guess it understands that. But that’s a very different problem from understanding Starcraft, and a much less difficult one at that.
And once the defense gets good enough, the attack portion of the AI should probably do the worker rushes much less often.
Of course, given that the AI has different constraints in micro-management and attention, and that the game is balanced given human skills, some things that would be totally bonkers to do as a human might work for the AI.
In Starcraft 2, it would be fantastic to individually control all 16 marines in your 2 medivac stim timing, pulling them back when they become the targets nearly instantaneously, perfectly shuffling them into/out of the medivacs, while focus-firing the defenses. Another example would be a reaper rush, a beastly AI could in theory micro 5-7 reapers individually, potentially breaking the game as human fingers/muscles/nerves have limits that CPUs and GPUs do not have.
The latter concern, about sheer speed and potentially limiting APM of the Starcraft 2 AI, is a very interesting one. It would be interesting to allow the AI to match the average APM of the Global Finals players, for instance, which might be around 300 actions per minute. If the computer was allowed to utilize 3,000 actions per minute, it could surely perform much greater feats of micromanagement.
The largest edge I see human players having vs the Starcraft AI would be something strategic, get a sense for how the AI plays then pull off a specific strategy which counters it. It may not work well in the current human-meta, but perhaps the AI will have its own meta that you can out-think it on?
Single unit pullbacks, medivac shuffling, and focus firing, all together, are commonly seen in pro-level matches. The guys are superhumanly insane. The current pro players can sustain 500 APM during battles. The only reason they don't do this even more is that they are usually fighting on two or three fronts at the same time, while managing base and unit production to the second.
The real limit of the players is the single viewport and the single instruction input pipeline. Managing dozens of units with those heavy restrictions is pure madness. That's the main difference between RTSs and MOBAs. MOBAs are defined by strategy and tactics, teams win and lose because of those two factors (at pro levels). A game like Starcraft II adds another factor: players can simply break down when they're not able to keep up with the game. Even at pro level, players can be overwhelmed by the amount of information they have to process in tenths of a second, over and over again, for 15-25 minutes, or they can get into a spot where there's too much to do and not enough time to do it.
AI has two advantages. It has better information channels, in the form of perfect vision, while humans have a single pair of eyes that have to scan through the screen. It also doesn't get tired over the course of the game.
Not to the level AI can do it. Did you see the video where zerglings dodge tank shot splash? a human could never do that
Perfect micro against siege tanks is impossible for humans, because the time between the beginning of the shot and the moment the damage is applied is seen by us as instant. The game features other units where that time is bigger, and pro-level games usually feature perfect micro on those instances (e.g. widow mines, banelings, disruptors, locusts, hellions...)
Consequently, limiting he AI's APM only adds a meta-game around APM planning that a human still would have a hard time to maintain.
Incorrect. It was with Starcraft 1, a custom map called Aeon of Strife (AoS).
https://web.archive.org/web/20101111183559/http://www.getdot...
It was specifically trained to do that (https://blog.openai.com/more-on-dota-2/) and the devs made edits to the model to improve it (https://twitter.com/Smerity/status/897959521661759488)
> Creep blocking can be learned from scratch. For 1v1, we learned creep blocking using traditional RL with a “creep block” reward. One of our team members left a 2v2 model training when he went on vacation (proposing to his now wife!), intending to see how much longer training would boost performance. To his surprise, the model had learned to creep block without any special guidance or reward.
a baseball "player" that can hit a homerun on every swing. or throw a 150mph fastball.
a basketball "player" that can make every single shot. you dont even need to defend that well when youre shooting 99%
a "golfer" that can get a hole on one in every hole
emulating human movement and agility will keep a few more sports safe, but not for long. im just sitting here waiting for the robot olympics
Sure we can make a basketball shooting machine with a high accuracy, but one that participate in a game, dribble, pass etc? Near impossible with today's technology.
The requirement that it'd be humanoid is just an artificial restriction that reveals you are not really asking for the best basketball machine that can be made.
https://www.washingtonpost.com/news/the-switch/wp/2015/03/24...
But because the agents were all trained together, they kind of operate as a unit.
Coordination
OpenAI Five does not contain an explicit communication channel between the heroes’ neural networks. Teamwork is controlled by a hyperparameter we dubbed “team spirit”. Team spirit ranges from 0 to 1, putting a weight on how much each of OpenAI Five’s heroes should care about its individual reward function versus the average of the team’s reward functions. We anneal its value from 0 to 1 over training.
(I never played dota2, that's all I know about the game ;)
You looked too early.
Might this be improved by forcing the AIs to send messages to each other at a constant interval? I feel like this would lead to the development of simple language built around intentions and help the Ai to consider its allies.
Communication in animals can be traced all the way back to cells sending each other messages to organize themselves. Cells began to have division of labor and work together, and as multicellular organisms increased in complexity, so did the complexity in their communication. But what would happen if you put these cells in charge of playing Dota and force it to send each other messages? Would the cells somehow develop a simple language built around intentions? Or does it still not understand the concept of intention?
Video games like Dota are extremely complicated and made specifically for human biases. We've obtained these biases over many many many generations of evolution. Machines are getting better at doing human tasks, but they still do not possess all the human biases (and I think it'd be quite scary if we actually get there). That's why these bots/algorithms/AIs can do simple tasks like identifying objects really well but they don't know what to do with them, yet.
Having machines engage in meaningful conversations with other machines and plan together is a much bigger problem to solve than having them farm up, buy items, and attack things.
This would likely benefit by some cherry picked training data, but I think AI could get good at communicating their state to each other. Star craft has too much entropy to make a simple example. Imagine the game was just, you pick a random number and your ally has to guess what it is. I think Ai’s could learn to communicate this using a similar training design.
It's also possible to cheat by spamming the same commands over and over to look "like a pro" in the stats and some people do this.
It's not unreasonable to have 200+ real APM in bursts, though. For example, at the beginning of games in Brood War, pretty much anyone competent is going to do something along the lines of the following in about 3 seconds:
click command center
build scv
select all scvs
mine a mineral patch
select a scv
mine a different mineral patch
select a scv
mine a different mineral patch
select a scv
mine a different mineral patch
click command center
set command center hotkey
set command center rally point
set camera hotkey
Step 1: Hotkey units on 1-9
Step 2: Slam face on keyboard numpad
Holding down a hotkey to build things (even if you don't have the minerals) also increases your APM.
Lots of players (especially the pros) will spam actions in the quieter parts of the early game to keep their fingers warmed up, or to start building a unit the millisecond they have enough minerals to afford it.
[1] pun intended
In all seriousness, this is pretty neat. Does anyone know if it already knows everything on the map, or if it has to discover units like the rest of us?
More detailed discussions at https://www.reddit.com/r/reinforcementlearning/comments/aioc... & https://www.reddit.com/r/MachineLearning/comments/aip7vu/d_d...
But it just won't be the same without someone telling me that I'm awful and my mother is of questionable morals.
... machine learning will probably pick that up too.
Example keyboard only game: https://youtu.be/cY0d2kHzrxY
I really hate marketing pre-news...
In halo, you have an AI that lives in your power armor and her name is Cortana. Cortana can sense what you see, hear, etc. Cortana is seen to be super smart on multiple occasions and it is clear to the player. This doesn’t bother anyone. It bothers me now, though. If all of this is true, the following would happen.
Cortana senses the presence of enemies faster than you. she hears them and is able to identify how many based on their foot steps. She becomes annoyed with how sloppy you are, how many enemies you didn’t even notice and how slow you are to respond to them. She suggests that she take over your power suit. You find yourself in a situation you can’t handle and are forced to give her the reigns. She jumps into action dispatching enemies with speed and accuracy that you’ve never seen before. She shoots enemies as they step out from cover. Your movements are so fast that you vomit. She wipes out army after army as you passively watch, bewildered.
As you go through the game, you are on a mission to unravel the mysteries of halo and find a way to survive. Cortana assesses the battle space and concludes that you need to travel east. She notices several patterns of ancient runes and directs you to investigate a hidden passage. Her assessments seem incredible and bizarre but they consistently prove to be correct and eventually she ends up making every decision.
At the end of the day, if Halo were real then it would be a story about a man who passively sat inside a suit and watched an ai save the world. He probably would die in the suit at some point due to experiencing high g forces during some adventurous fight or maneuver and it would be a story about an ai who saved the world with a dead guy in the suit. It doesn’t make you feel warm and fuzzy like halo is supposed to, does it?
I take it you haven’t played Halo Reach :)
It is therefore only through deep, contemplative and unintuitive ruminations that one can come to understand that AI is going to end human society as we know it, and that the result will be very unpleasant. This is not a guess, it is the truth. People have a huge amount of difficulty seeing this. I thought the example of halo and Cortana would make it easier for people to understand. I could have written it better but why bother? Nobody ever seems to get it.