Hacker News Reader

Comment by wpasc | Hacker News Reader

I wonder how Starcraft2 compares to Dota 2 in terms of difficulty for the AI. Moreover, I'd be interested to see how OpenAI and DeepMind's techniques, models differ in approaching these games.

View on HN

The main difference would come in the simultaneous control of all of the different units in Starcraft. Both MOBAs and RTS share many characteristics, since Warcraft 3 was the origin of MOBAs to begin with. For instance, micro, unit pathing, lowground highground, fog of war, spellcasting, etc. However, in a MOBA there is more emphasis on team-play, on the interactions of 5-hero team compositions, which is decided upon at the start of the game. There is also itemization, which is much more stylistic than anything that can be purchased in an RTS.

In Starcraft 2, it would be fantastic to individually control all 16 marines in your 2 medivac stim timing, pulling them back when they become the targets nearly instantaneously, perfectly shuffling them into/out of the medivacs, while focus-firing the defenses. Another example would be a reaper rush, a beastly AI could in theory micro 5-7 reapers individually, potentially breaking the game as human fingers/muscles/nerves have limits that CPUs and GPUs do not have.

The latter concern, about sheer speed and potentially limiting APM of the Starcraft 2 AI, is a very interesting one. It would be interesting to allow the AI to match the average APM of the Global Finals players, for instance, which might be around 300 actions per minute. If the computer was allowed to utilize 3,000 actions per minute, it could surely perform much greater feats of micromanagement.

The largest edge I see human players having vs the Starcraft AI would be something strategic, get a sense for how the AI plays then pull off a specific strategy which counters it. It may not work well in the current human-meta, but perhaps the AI will have its own meta that you can out-think it on?

> In Starcraft 2, it would be fantastic to individually control all 16 marines in your 2 medivac stim timing, pulling them back when they become the targets nearly instantaneously, perfectly shuffling them into/out of the medivacs, while focus-firing the defenses.

Single unit pullbacks, medivac shuffling, and focus firing, all together, are commonly seen in pro-level matches. The guys are superhumanly insane. The current pro players can sustain 500 APM during battles. The only reason they don't do this even more is that they are usually fighting on two or three fronts at the same time, while managing base and unit production to the second.

The real limit of the players is the single viewport and the single instruction input pipeline. Managing dozens of units with those heavy restrictions is pure madness. That's the main difference between RTSs and MOBAs. MOBAs are defined by strategy and tactics, teams win and lose because of those two factors (at pro levels). A game like Starcraft II adds another factor: players can simply break down when they're not able to keep up with the game. Even at pro level, players can be overwhelmed by the amount of information they have to process in tenths of a second, over and over again, for 15-25 minutes, or they can get into a spot where there's too much to do and not enough time to do it.

AI has two advantages. It has better information channels, in the form of perfect vision, while humans have a single pair of eyes that have to scan through the screen. It also doesn't get tired over the course of the game.

> Single unit pullbacks, medivac shuffling, and focus firing, all together, are commonly seen in pro-level matches. The guys are superhumanly insane. The current pro players can sustain 500 APM during battles. The only reason they don't do this even more is that they are usually fighting on two or three fronts at the same time, while managing base and unit production to the second.

Not to the level AI can do it. Did you see the video where zerglings dodge tank shot splash? a human could never do that

That zergling video depends on a zero reaction time, it's true that's an advantage of AI. No matter how high the APM, if the reaction time is high enough, some things just cannot be done.

Perfect micro against siege tanks is impossible for humans, because the time between the beginning of the shot and the moment the damage is applied is seen by us as instant. The game features other units where that time is bigger, and pro-level games usually feature perfect micro on those instances (e.g. widow mines, banelings, disruptors, locusts, hellions...)

> Even at pro level, players can be overwhelmed by the amount of information they have to process in tenths of a second, over and over again, for 15-25 minutes, or they can get into a spot where there's too much to do and not enough time to do it.

Consequently, limiting he AI's APM only adds a meta-game around APM planning that a human still would have a hard time to maintain.

> since Warcraft 3 was the origin of MOBAs to begin with.

Incorrect. It was with Starcraft 1, a custom map called Aeon of Strife (AoS).

https://web.archive.org/web/20101111183559/http://www.getdot...

Yes, although MOBAs were popularized with War3, specifically DotA All-Stars became utterly dominant and more popular than the base game.

When deepmind announced the project, they noted they were going to limit the machine's APM to something human-like

A problem I see with limiting APM to the same level as a human is that for that APM, a human will move units tightly packed together, while a computer will likely move a bunch of units at different ends of the map

The conclusion would be to patch the game's unit control with the new AI. That would augment human play and the implementation would need to be careful not to become the equivalent of auto-aim in first person shooters.

https://ai.stackexchange.com/questions/3814/whats-the-differ...

It is somewhat disturbing that the AI learned to creep-block on its own.

Where did you read that it learned "on its own"?

It was specifically trained to do that (https://blog.openai.com/more-on-dota-2/) and the devs made edits to the model to improve it (https://twitter.com/Smerity/status/897959521661759488)

That was from 2017 1v1 bot. OpenAI Five apparently learned it on its own. From: https://blog.openai.com/openai-five/

> Creep blocking can be learned from scratch. For 1v1, we learned creep blocking using traditional RL with a “creep block” reward. One of our team members left a 2v2 model training when he went on vacation (proposing to his now wife!), intending to see how much longer training would boost performance. To his surprise, the model had learned to creep block without any special guidance or reward.

It’s going to get even creepier for you then. I’d be very surprised if a decade from now any human player has a chance of beating a state of the art AI player in pvp games.

that's not creepy at all. there's many sports today where a machine specially built for that task would dominate any human from ever winning.

a baseball "player" that can hit a homerun on every swing. or throw a 150mph fastball.

a basketball "player" that can make every single shot. you dont even need to defend that well when youre shooting 99%

a "golfer" that can get a hole on one in every hole

emulating human movement and agility will keep a few more sports safe, but not for long. im just sitting here waiting for the robot olympics

You are really underestimating the difficulty in making humanoid robots that actually work well.

Sure we can make a basketball shooting machine with a high accuracy, but one that participate in a game, dribble, pass etc? Near impossible with today's technology.

It just needs to be tall, move with some speed, and be able to receive passes from other players; essentially a easier to hit hoop that can annoy the opposing team by getting in the way.

The requirement that it'd be humanoid is just an artificial restriction that reveals you are not really asking for the best basketball machine that can be made.

It'll have to follow the rules of basketball, though, which were written for human players and refer frequently to 'hands', 'feet', 'steps', 'arms', 'legs', and so on.

More interesting would be humanoid robots that could win at these sports. The state of the art today is a long ways away from being able to compete in any of the listed sports. Baseball is probably closest because with sufficiently good pitching and hitting you could win, which would avoid the much more dynamic problems of fielding, running bases, etc.

The first two, maybe, but there's no way we could build a golfing robot that gets a hole in one on every shot.

In an indoor field it's totally possible. Standard golf is affected by weather, though.

at the very least the machine will 1-putt from anywhere on the green

Reminds me of this woz quote an associate and I say to each other on occasion when creepy AI comes up: "The future is scary and very bad for people" which we shorten to "The future is scary and very bad." :)

https://www.washingtonpost.com/news/the-switch/wp/2015/03/24...

state of the art AI has been unbeatable in FPS since the first aimbot was created.

I'm interested to see how advanced AI can get at Spy Party

I would think SC2 is harder than Dota 2. In Dota you control 5 units while SC2 you can build hundreds.

Technically, in Dota you only control 1 unit. That being said, I don't know if OpenAI's algorithm had 5 independent agents controlling 1 unit each, or 1 agent controlling 5 units.

I believe it's five identical agents: https://blog.openai.com/openai-five/

But because the agents were all trained together, they kind of operate as a unit.

Coordination

OpenAI Five does not contain an explicit communication channel between the heroes’ neural networks. Teamwork is controlled by a hyperparameter we dubbed “team spirit”. Team spirit ranges from 0 to 1, putting a weight on how much each of OpenAI Five’s heroes should care about its individual reward function versus the average of the team’s reward functions. We anneal its value from 0 to 1 over training.

It would be fun to see the AI players spamming the ingame chat to communicate with each other in its own language :)

A truly superb AI would guilt the support AI into buying wards by constantly whining about it in chat. It would also ping the minimap when one of its teammates makes a mistake. Finally, at the end of the match, it would pronounce "bad team" if it lost.

(I never played dota2, that's all I know about the game ;)

In the last model they trained they shared some part of a layer for all the agents. So it was like a one agent trained with shared knowledge of the rest.

OpenAI Five had a separate agent per unit, no explicit coordination.

This is incorrect. They shared neural net activations which is explicit communication.

at the same time that could be seen as an advantage for the machine, as multitasking is much easier for a massively parallel computer not limited by muscle movement or vision than for us. we'll see. depends on the usual stuff like limited apm ofc

I wonder if they stuck with their original game playing API that takes in some "visual" features. In that case it would be "limited" by vision.

in dota there are 10 players, so you have to coordinate. apples to apples this would mean 5 indepdendent bots per team that have to learn that. there are also summons to control and elaborate spell interactions.

Reply on news.ycombinator.com