StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer
arxiv.org
arxiv.org
I am not very surprised that machine learning has been able to successfully execute micromanagement. In my opinion the macro decisions are more interesting, since they typically require higher level reasoning (and comprehension of what the opponent is doing). We have yet to see an AI system that can successfully execute macro strategy when playing against a human opponent.
I watched a pro game just the other day where one of the players noticed that their protoss opponent had a late timing on their second pylon in their main base - as in, it should have been there, but wasn't. That protoss player has a well-known penchant for early game stargate harassment, so the other player read this situation and reacted by delaying their first expansion and instead producing early-game air defenses.
As another example, take unit composition. If a terran player is running a bio-based army (marine, marauder, medivac), what sort of units should zerg produce? Infestors are usually a good choice in this matchup. On the other hand, mass infestors will be much worse against a primarily mech-based terran army; instead you might see many vipers on the field (which, vice versa, wouldn't be great against a bio army).
Pro players are essentially always scouting what their opponent's doing because the macro decisions can make or break the game. The general sentiment I've seen that is good macro skills will push you far up into SC2 leagues, and it's not until diamond/masters that micro starts to really matter.
I watch a fair bit of pro play, and also play enough to bounce between low diamond and high platinum on NA. It is almost always my macro that wins or looses games.
Most people at my level wouldn't even have consistent enough play to be able to notice the difference between well and poorly micro'd games. But if they miss an upgrade, or build the wrong comp, or worse, don't expand, that is the win/loose for 98% of games below masters.
Which all around sucks for me, because my micro is on point :)
Some good but not quite star quality players like Ketroc deliberately ignore the meta. Ketroc plays with loads of Ravens, he did this when people said Ravens were strong but hard to use, he still did it when the accepted wisdom was that they were now useless, and last time I looked he was still at it after yet further balance changes to SC2. How he uses them changes, but he likes Ravens and they never stopped being viable.
Games like Chess have meta too. A year ago humans believed that materiel was essential and strong position made up only for small differences like losing a pawn or two. Then Google's Alpha Zero AI says hold my beer, sacrificing loads of my pieces for positional advantage is fine, because I can use that advantage to pin your extra pieces and still mate you away.
I think that SC2 and BW differ in this respect because SC2 has been constantly updated throughout its competitions to make more strategies and units viable. The last balance patch for BW was two years after release in 2001 (although a recent bug fix is said to have slightly buffed valkryies, there is some dispute about whether this actually pans out in practice but I haven't looked into it).
The basics (Spend your money, don't queue units, don't get supply blocked) are trivial. Executing them perfectly for 20 minutes, during a stressful game, is impossible. The best players in the world can't do it.
Thus, strategy is the widest tent, and someone like sOs shows the truly massive number of strategies that can be employed. In contrast, macromanagement does indeed have a smaller decision space, since really the "decision" with macro is pre-defined. Make your economy and production as efficient as possible. There may be slight variations in how its done, especially in Brood War where the economics are much more nuanced than SC2. Nevertheless, if you watch a variety of pros macro out mass marine-marauder-medivac, they will usually have very very similar expansion timings, identical numbers of barracks, and any changes in the timings/build orders is strategic, not macro-related. There may be "on the fly" macro adjustments due to harassment damage, however strategy is far far more fluid of a category than macro.
When it comes to micro and decision space, however, I believe that it is mostly a matter of execution and not decision. We've seen perfect blink micro stalkers, dragoon AI bots, and I think it'd be far easier to take down top players in a 1v1 micro fight since computers are not limited by mouse accuracy, emotional concerns, stress, fatigue, and a number of other factors which affect your hands' abilities. In contrast, a top player should have a much better edge over a computer in strategic realms, since strategies must be tailored to the expected opponent's strategy and mind games heavily factor into this equation.
God, what a beautiful game.
If you ignore the part where you have lots of choice, then there's little choice, indeed :)
And yet, something similar may have been said about Go, Chess, Checkers, Poker, Jeopardy, or many other games before this. If enough talented minds and powerful devices focus on Starcraft, how would humans compete? Does anyone here think top pros would win in a best of 7 or a best of 11, for instance? Which top pros, and BW or SC2? To me, I just think it'd be arrogant to assume SC is different from all the other realms in which humans have been bested by machines.
From a human perspective, Starcraft doesn't seem "harder" than Chess or Go, but it has more fluidity and ambiguity, both things computers struggle with. And the ruleset, as it were, is VASTLY more complicated.
The main advantage AI can exploit is "infinite micro" units. There are certain units in SC2 that are primarily limited by someone's ability to pay attention to them. Terran Reapers, for instance, in early patches could dance back and forth just within range Zerg Roaches, but out of range of being counterattacked (though the reapers would quickly die if player stopped paying attention). This required such fast movement that it was almost never employed except by professionals during the early game when not much was going on.
If an AI is not limited by the number of actions a person can take, these "infinite micro" units become extremely powerful. In particular, most of these types of units are within the Terran race and most powerful in the early game. The best AI in the near future will likely focus on early all-in attacks using these sorts of units (primarily reapers, widow mines, hellions, and medivacs). The human player would have to rely on defensive structures that would put them at a major disadvantage and then catch up in the mid-late game, relying on the fact that the AI gets worse as the game goes on and becomes less predictable.
Another thing to note is that Terran strategy relies quite heavily on killing the enemy workers in the early game, but the inbuilt controls for targeting workers are quite clunky and prioritise military targets. Players can target individual workers manually, but generally do so with their whole army at once, while an AI could select the perfect number of units required to 1-shot a worker, give them the command, then select another group, and so on.
If you want to stay "human-comparable", it sounds as if it wouldn't be hard to artificially impose those restriction on an AI - e.g. you could simulate a "viewport" that the AI can move around and restrict the micro engine to only take inputs from within that viewport, etc.
(Assuming the AI could make use of that advantage anyway. If I remember correctly, there is a lot of research going on that uses "attention" itself as a primitive in AI architecture. So depending on the architecture, AI might not always be able to make use of that advantage.)
On the other hand, an AI that does use infinite-micro strategy might be useful for UI research. E.g., you could pit it against a human player and use the human player's performance as a benchmark for UI improvements.
The information is only imperfect for a human but an IA can have a way better access to information: - if you scout you opponent mineral line and click on the minerals, you know exactly how much minerals he's harvested since the beginning of the game. If you scout your opponent army and building, you know exactly how much he spent. Then you can deduce if he has something hidden somewhere or in production. Of course this is impossible to do for a human, but really easy for a computer. - scouting is easy but requires a lot of actions, an IA could have one mutalisk wandering over the map the whole game without being killed, which is impossible for a human unless you dedicate full time on this.
Imho, beating humans to StarCraft isn't a big deal in itself, but it would need a lot of work to build the AI to deal with the game's complexity. But it's merely domain specific work that has little to no interest on AI in general. Creating an AI that trains itself how to be good at StarCraft will take much more times and effort, but at least it will provide value outside of the specific problem.
This is definitly not impossible for humans. High level SC players are able to keep track of the mineral and gas spendings of their opponent in the early game (After the early game there is usually no complete information to calculate this). Just seeing the timing of taking the gas geysers gives you information about the earliest possible time a certain unit/building can be produced.
Of course a AI can scout more and take more variables into account but humans are surprisingly good at this too.
> Imho, beating humans to StarCraft isn't a big deal in itself, but it would need a lot of work to build the AI to deal with the game's complexity. But it's merely domain specific work that has little to no interest on AI in general.
The same thing was said about early chess AIs and the Alpha GO project. In the end both lead to major advancements in AI. If you solve the domain specific problem starcraft you could easily apply the gained knowledge to similiar problems.
You can't get much more scouting than that because there is a cost associated with scouting, either because you lose a worker's time or because you spread out your army.
I agree but I was mainly talking about the time it takes to perform the scouting. A AI could scout 5 different places at the same time which is pretty hard for a human to do without falling behind elsewhere. I know SC pros are really good at multitasking but at some point the AI is just superior. If this gives the AI a significant advantage is a different question.
Afaik, chess AI (I mean, modern ones like deep blue) had little to no wide impact on AI. They had a lot of impact on chess though.
But it's way easier to model a chessboard than a game like StarCraft, the domain specific work is just orders of magnitude bigger.
Apart from the pure technical side (which I am no expert in so you might be right) the impact on public attention and awareness for AI was huge for those projects. It's important that "normal" people get a feeling for AIs and what they are able to accomplish.
But even if you want to stay in RTS land, one of the more recent titles that comes to mind where micromanagement is not as important and strategy/planning is very important is Ashes of the Singularity. Spoiler: the game may seem "slow" compared to Starcraft but that's also part of reducing the importance of high APM.
as an example see the zerg vs tank below.
I understand the idea of code manipulation but what might be better is just having human programmed AI's fight each other (which I am fairly sure is going and can be viewed live but I do not have the link handy.
The problem with starcraft is that its like chess - for a lot of players it really DOES get stale pretty quickly, with only one or two interesting moments in a game (assuming no early blunders). The games that are interesting involve near perfect play for 10-20 minutes and then frequently one split second decision. Macro changes or new strategies are quickly picked up and interated on, and I think thats what the hardcore still stick around for
Rate limiting a brute-force AI would make it trivially bad.
https://www.youtube.com/watch?v=0EYH-csTttw
Also wondering what the interface is - are they creating a virtual mouse/keyboard to enter inputs, are they playing in some sort of headless mode? How do they deal with recognizing units?
Better link: https://clips.twitch.tv/BusyDrabFennelRedCoat
When I pick targets I prefer to kill units that deal the most damage or have low HP first.