Comment by wamatt | Hacker News Reader

Comment by wamatt | Hacker News Reader

Generalized reasoning in strategy games AI is an especially difficult problem for machine learning. IRL, top StarCraft players routinely model their opponents mental states and psychology to create an edge.

So perhaps it's worth pointing out, that this paper specifically addresses a sub-problem of Starcraft play, micromanagement ('micro') [1]

The game engine runs at 24 frames per second. (As an aside, 'frames' in this context likely does not map to physical FPS of the display).

>We ran all the following experiments with a skip_frames of 9 (meaning that we take about 2.6 actions per unit per second).

The research team found that attempting to move at a superhuman pace (eg one action every frame), resulted in a subpar performance and hyper-parameterization indicated 2.6 to be an ideal action per second.

In context, this translates to an APM of 156. Or, roughly half that of professional Korean e-athletes. [2]

[1] https://en.wikipedia.org/wiki/Micromanagement_(gameplay)

[2] https://en.wikipedia.org/wiki/Actions_per_minute

"The researchers found that attempting to move at a superhuman pace (eg one action every frame), resulted in a subpar performance."

Moving at extremely fine-grained timesteps can make learning much more difficult, because now a reward arrives millions of timesteps delayed rather than hundreds or thousands. It's like trying to teach a NN to compose piano music by starting down at the 1ms raw audio level. This is part of why audio synthesis was so difficult up until recently with DeepMind's WaveNet. In theory, being able to move every frame should enable extremely superhuman performance, but in practice, you can't learn your way there. So often people will chunk data to make it easier to learn the higher-level concepts: operate on words, rather than characters, for example.

Why not go the other way and decrease the actions per minute so you learn the overall point of the game , And with each game the actions per minute increases.

That sounds like an interesting research angle. The thing about AI research is there are so many open ends there are essentially unlimited research options. If you can pose it as a problem and identify a reasonable programming approach then you have an avenue for AI research. Deep Learning isn't the end of AI research. It is the beginning.

Or maybe extend the traditional categories of macro and micro with another one, call it 'nano'... the micro agent indicates where each unit ought to be in 9 frames, and the nano agent figures out how to take them there. Since the timescale is so short, the agent could brute-force enumerate possible moves to some extent and figure out which is optimal, like chess AI. Or use a separate network.

I guess that's inelegant when a deep network already has its own concept of fine-grained versus coarse-grained layers, and should be able to do this on its own with the right training method.

APM numbers are actions-per-minute, not the actions-per-unit-per-minute measurement in which the AI's performance is reported. It's not clear how the two map to each other -- the AI may well be exceeding human performance when multiple unit groups are considered.

That's a good point and not really correct to conflate the two.

It would then seem to suggest, that forces with more than say, 5 units (780 average sustained APM) and above, would likely be getting into super human territory.

I am not a gamer... but I have to ask. 780 APM is possible with human players? 13 actions per second? I can't come close to clicking my mouse that fast. I don't think I can blink that fast.

Yes, it's possible. Bear in mind that not just clicks count though. July managed a peak APM of 818 in Stracraft: Brood War in an official game. The average APM of a match is usually between 300 and 500, also depending on the race that is played.

A lot of that is 'spamming', where they keep their hands moving and doing things even if they're not particularly meaningful, just so they can keep the pace of the eye/brain/hands feedback loop going.

Maybe early on it is spamming, but later on in the game most actions because useful actions. Having said that, some APM measurements only tracked effective APM (EAPM), which was clicks that actually achieved some result rather than merely selecting a unit.

I think that may be high for an average... and to give an idea of how much APM is spam, there's that Stork vs. Idra replay from the World Cyber Games where Stork demolishes Idra despite using 70 APM to Idra's 300+. Of course, Protoss takes the least APM but Stork may have been the least spammy player.

There are hotkeys. there's a big map you can scroll around, but you can bind a specific view of a map to f1-f4. you can get a lot of actions just rolling through those, looking for specific events. Is the attack coming from over here? is the base still collecting resources? stuff like that.

You can also hotkey individuals or groups, 1-8. Again just rolling through those will get you a fair amount of apm.

That said, 780 is peak for world class players. You probably can't do that, ever, because by the time you put in the years of effort, your reflexes will have decayed.

On the other hand, brood war was a subtle game. APM is not the be all end all of winning. Getting proficient with a few aspects of the game can make you pretty good pretty quickly. Heck, just looking around the map like i mentioned above will help. Some people are amazing controlling a few units like a surgeon. Those units are practically unstoppable. But they miss the big picture. Scanning the map lets you distract, delay and ultimately minimize the value of those unstoppable units.

I found this video demonstrating ~300 APM, which is quite insane already. I think >500 APM is probably burst speeds, not sustained.

https://www.youtube.com/watch?v=zmYhX8fjmo8

Reply on news.ycombinator.com