Grandmaster level in StarCraft II using multi-agent reinforcement learning

I'm sure that the AlphaStar effort absolutely dwarfed everything that came before in terms of investment (as I'm sure AlphaGo did for Go as well) but based on my reading I'd also bet that Starcraft has likely seen the most continued AI research effort of any imperfect information, real-time game over the last decade - I think that's also part of what made it appealing for DeepMind over Dota 2 or any of the other options in this class of game.

[1] https://arxiv.org/pdf/1909.02682.pdf [2] https://arxiv.org/pdf/1907.09467.pdf [3] https://arxiv.org/pdf/1906.12266.pdf [4] https://liquipedia.net/starcraft/SSCAIT

It doesn't matter how much money or interest we have, but right now it isn't technically feasible to build a 36000 km tall crane (also known as a space elevator). Humanity simply couldn't get it done even if we poured all our current resources into that project. It isn't physically impossible, but such a behemoth has requirements that current materials science cannot meet. Building tall cranes generates useful know-how for a space elevator, but it's a really different problem, not just a matter of resources.

Following the analogy, a general purpose AI simply isn't a bigger Deep Blue or AlphaGo; it's probably something different that requires knowledge that we currently don't have. Sure, building Deep Blue and AlphaGo most likely generated part of that money, but that doesn't mean we have everything we need.

You can argue that is simply also money and interest, and that's true, but it's not in the same as building a 10 meter crane and a 20 meter crane.

On a practical level, this is not true. There are different algorithms, different architectures, different hyperparameters required for each of these problems, and often for each subdomain within each of these problems, and often for each specific instance of these problems. It's difficult to draw any kind of holistic picture that combines all of the individual advances in each of these problem instances; that's why progress in AI is so hard to measure, and why a statement like "each of these toy problems...brings us closer and closer to solving the 'real problems'" is probably a bit too coarse-grained to be fair as well.

> Anytime a next task is solved, there is a crowd saying it's not a "real AI" and that scientists are solving "toy problems". Both statements are totally true. But the underlying substance is that each of these toy problems is of increasing complexity and brings us closer and closer to solving the "real problems"

I wonder if this is true. This belief may seem like common sense, but it's not obvious to me that domain-specific problems must generalize to General AI ("real problems") or even bring us closer to it. That is, it's not evidently true that many small problems will eventually lead to a general solver of everything (or to human-like intelligence). Or to say it in yet another way, it's not obvious to me that human-like intelligence is the sum of many small-problem-intelligences.

Again, common sense may lead us to believe this, and maybe it's true! But I think this conclusion is far from scientifically evident.

As another poster said these are all tasks performed by different systems. For chess and Go AI it's Deep Reinforcement Learning with Monte Carlo Tree Search. For image recognition it's Convolutional Neural Networks. Importantly, these systems are very task-specific. You won't find anyone trying to beat humans at games using CNNs, for example, or using Deep-RL to do text recognition. Far from "a few creative tricks" these are systems that are fundamentally different and are not known to generalise outside their very limited domains. They're one-trick ponies.

The OpenAI paper on "dexterous manipulation" reported learning to manipulate one cube, the same cube, always, after spending a considerable amount of resources on the task. It was a disappointing result that really shouldn't be groupwed with CNNs and Deep-RL for game playing. The level of achievement does not compare well.

>> Anytime a next task is solved, there is a crowd saying it's not a "real AI" and that scientists are solving "toy problems".

This used to be the case a decade or more ago. In the last few years the opposite is true. The press is certainly very eager to report every big success of "AI"- by which of course is meant deep learning.

>> 6 years ago we were able to solve some Atari games from pixels. Today, that feels like a trivial exercise compared to modern techniques

6 years ago DeepMind showed superhuman performance in seven Atari games with Deep-RL (DeepQN in particular): Beam Rider, Breakout, Enduro, Pong, Q*bert, Seaquest and Space Invaders. Since then more Atari games have been "beaten" in the same sense, but many still remain. I'm afraid I can't find references to this but I've seen slides from DeepMind people a few times and there is always a curve with a few games at the top and most games at the bottom, below human performance. There are some games that are notorious for being very difficult to solve with Deep-RL, like Montezuma's Revenge which was claimed to be solved by Uber a couple of years ago however this was done using imitation learning, which means watching a human play. The result is nothing like the result in Go, which remains the crowning achievement of Deep-RL (and its best buddy, MCTS).

Bottom line: Atari games remain anything but a trivial exercise.

And the architectuers that play Atari do not perform as well in Go or chess, say. You are mistaken that it's simple to train the same system to do all of those things. The AlphaZero system that played Go, chess and Shoggi well enough to beat its predecessor (you will excuse me that I don't remember which incarnation of Alpha-x it was) had an architeture fine-tuned to a chessboard and pieces with discrete moves, so it would not be possible to reuse it to play Starcraft, say, or even tic-tac-toe. The cost to train AlphaZero is also very high, in the hundreds of thousands of dollars.

Yes, a human is still needed to decide which approach to use, but we are slowly approaching a world where building an AI becomes easier and faster. It will become absolutely irrelevant that a single AI can not play all games, because whenever a new game comes out, someone will be able to build a superhuman AI on it within a week/month.

The brains of an AI are also transferrable. Built a superhuman AI? Send it to a friend! Compare it to a human, who would need to spend enormous amounts of time to transfer their game-brain to someone else. If you want, you can bundle all those AIs into one and pretend it can play any game.

"It's amazing that most people here don't understand that human performance in all computer games relative to AIs is largely irrelevant"

Also I think you overestimate the AI/IT knowledge of these top players that they're consulting. I have great respect towards them, but they're not renaissance men [2] who both play 10 hours of StarCraft per day and also know the subtleties of how computers work, not to mention bleeding edge AI. You can watch the previous showmatch [3] to see how DeepMind people lack knowledge of StarCraft and how the pro players they're consulting lack understanding of the AI and both are learning new things live on air. Its obvious that their cooperation is bearing fruit as they spend more time talking to each other, as evidenced by the new APM limits. However I'm willing to bet that they would reach many more good conclusions if they just continue working together.

--

[1] https://en.wikipedia.org/wiki/Chaos_engineering

[2] The problem with both parties (the pros & deepmind) is that they're so overspecialized. I'm nowhere near as good as them at their respective fields, but I am a professional programmer and diamond in StarCraft II. In addition I've built StarCraft II AI myself, although with different goals related to finding optimal strategies.

[3] https://www.twitch.tv/videos/369062832

Is it because they expect to lose workers to harassment? Or is it so that they can saturate new expansions quicker?

> People oversaturated in wol and hots because you didn't expand a lot in these games and bases have a lot less minerals in LOTV, not because of some kind of lost knowledge like some comments seem to think here.

[...]

> Meanwhile alphastar was going 2 gate robo staying on one base making a fuck ton of probes to take a super late natural, it was bad, and anybody calling it the future really bothered me.

Alphastar has not introduced anything new. Its play is strategically poor.

This is off-topic, but I was thinking about this the other day - couldn't AI be used to balance a competitive game in this sense? Imagine that the AI becomes so good that human players very rarely win against the AI in a best of 7 series. Then we find out that AI doesn't ever build a specific unit. That would be a pretty strong indicator that said unit isn't good enough compared to the rest.

Warcraft 3 and broodwar were much more micro-oriented than sc2 (because ai was much worse for example if you let dragoons to walk to their target on their own they spend half the time fighting broken pathfinding while the enemy shoots at them :) This almost never happens in sc2, and microing units below master league is usually detrimental to the game).

I think the micro vs macro is not the full image, and sc2 lets player express themselves on what i would call mediumgame (something that is between buildorders and microing - like deciding when to expand, what tech-switch to make, how to position your army in between fights, etc).

There's lots of decisionmaking that isn't APM-limited but thinking-time limited in SC2, and it's not as codified as buildorders. But you can certainly simplify the games to "archon immortal vs bio terran", and then it's very repetative.

It's kind of like competing against a gorilla in boxing chess. Just because a gorilla is dominant doesn't mean boxing chess doesn't require chess skills, only that a gorilla doesn't need them.

Raw, spur of the moment, innovation probably isn't a good idea generally, as you'd be doing something with no practice. People do have a fairly big bag of tricks from older tactics that are less common these days.

Designing a game that is a fun and balanced experience whilst also encouraging true innovation is a difficult problem. Every game ever has this problem, putting thousands of minds on a problem and letting them confer about their results usually solves all the low-hanging fruit in about a week. Creating a game where you only play to share a new idea makes each match feel like a dice roll.

It’s just that a lot of that strategy is irrelevant unless you have amazing mechanics - and the difficulty of those mechanics means that you can’t as easily think, plan, or adapt in-game (because your brain is busy)

A big part of the game is thinking about “how do I win against somebody who does that...” between games

That humans cannot reliably perform these actions because of the limitations of our corporeal form means that Alphastar has an advantage over a human player. Limiting APM isn't enough.

One of the videos I watched compared APM (Actions Per Minute) with EPM (Effective actions Per Minute). AlphaStar always has them nearly identical, which would be (according to him) basically impossible for humans.

Scrolling screen is a major action to gather info and control units.

If one can control near edge units without scrolling it gives more stable view and lower chance of making mistakes.

This is not true. Top-level human players take into account who their opponent is, what builds they've used in recent games, their proclivities, strengths, and weaknesses. In a 7-game match they'll intentionally use builds which (in order to deceive) appear the same but have very different effects.

AlphaStar has better strategy than other AIs, but that's a low bar. My young son has better strategic appreciation of the game, and easily points out the difference between good strategy and AlphaStar's next-level micro.

The configuration of control groups is visible in replays; AlphaStar's replay data does not have any control groups in it.

Though, via its API, it is able to select arbitrary groups of units from anywhere on the map.

Personally, I would have loved to see it work through control group management because I think that is something that is important and steals attention away from humans. But probably to the researchers, it is just annoying data to model that doesn't get to the "core" of StarCraft.

Of course to any reasonable person a consistent 5 ms is obviously vastly superior to once in a blue moon lottery-winning luck.

Sounds like it's the same deal here, where the designers made some effort to level the interface playing field, but also left in some advantages like instantly being able to select individual units (or whatever it was the other players felt was not humanly possible to do). Probably because, at some level, they really want their system to win.

Strategy and tactics are a function of constraints. If an unconstrained computer can beat a constrained human, have we really shown that its strategy and tactics are better? To do that, you'd have them to play under the same constraints. Or at least have the constrains be close enough, which is what people are debating.

They didn't say anywhere they scored a win against the top 10 either.

To elaborate: to be among the "top 0.15%" of 90k is place ~14 (13.5) in the worst case, so nowhere near the "commanding" abilities mentioned by GP

Unless you're Serral and your observers magically come out of nowhere and cover every single pixel of your base.

Flash had a 70% winrate overall and had time periods of 90%+ winrate.

From what I'm reading in the paper, it sounds like there is some custom interface in play:

>AlphaStar can target locations more accurately than humans outside the camera, although less accurately within it because target locations (selected on a 256x256 grid) are treated the same inside and outside the camera.

It's really hard to parse what such statements mean. The fact that someone who is cited as a co-author of the paper approved the interface as "fair" isn't particularly reassuring.

Also I wonder how their "camera-like interface" works with tactics like fly a building above units to make them harder to target.

>During training we used three agent types that differ only in the distribution of opponents they train against, when they are snapshotted to create a new player, and the probability of resetting to the supervised parameters.

So exploiter agents aren't fed a specific strat, instead they discover the weak spots in the same way as the main agent tries to win. The GANs similarity is there.

Just about sums up business school.

these 3 pieces of info already allow you to create a strategy framework