Grandmaster level in StarCraft II using multi-agent reinforcement learning | Hacker News Reader

Grandmaster level in StarCraft II using multi-agent reinforcement learning

deepmind.com

42 pointsJyaif18 comments

Its not clear all the actions they've taken to limit themselves to human input. Some players feel that the EPM (effective actions per minute) of AlphaStar is enormously higher then a human's during a battle. This is likened to a more advanced aimbot where it teaches human's nothing and has an unfair advantage. Alternatively, alpha go has advanced the state of human go playing without feeling like it cheats.

AlphaStar is impressive but I don't think AI has quite dominated the game of starcraft as convincingly as it has go.

It seems that the superhuman EPM was more than just a feeling, because human players were able to correctly identify DeepMind's AI from its actions:

After 50 games, however, DeepMind hit a snag. Some players had noticed that three user accounts on the Battle.net gaming platform had played the exact same number of StarCraft II games over a similar time frame — the three accounts that AlphaStar was secretly using. When watching replays of these matches, players noticed that the account owner was performing actions that would be extremely difficult, if not impossible, for a human. In response, DeepMind began using a number of tricks to keep the trial blind and stop players spotting AlphaStar, such as switching accounts regularly.

From the Nature article:

https://www.nature.com/articles/d41586-019-03298-6

Although hiding AlphaStar's identity was part of the experiment, it also seems a bit sneaky to try to hide the smoking gun of impossible actions.

The main impossible action that serves as an AlphaStar giveaway is that it doesn't use control groups. That by itself doesn't suggest an unfair advantage, though. It's just noticeable in replays. The APM still seems to be in line with high-level players.

If I remember correctly (from other games), control groups means that you can select a group of units and assign a number to them (Ctrl+n?) so you can then issue commands to them as a group.

I think that a human player's game would be significantly slowed down if they had to play without control groups. Accordingly, the ability to control units without having to assign them to control groups must be a very big advantage for the purpose of micromanaging units. It would be like playing a turn-based game when everyone else was playing in real time.

True, you might be right. It might be using an abstract notion of control groups internally (learned through watching and playing so many games), capped at the typical control group cap, in which case maybe it's not an advantage. But maybe it is able to have much more fine-grained control over every single unit, in which case it probably is an advantage.

It was also selecting buildings that for a normal player would either be out of the isometric field of view, covered by ui elements, or only barely on screen like clicking 1 pixel specific in the corner.

That's a big problem, and if true would probably invalidate the results. I would guess that AlphaStar was giving unrealistically complicated orders, or more likely tons of orders on opposite sides of the map simultaneously, a feat that would be all but impossible for a human.

You're overstating the issue here. Deepmind wasn't forced to control units using the standard control groups. My understanding is they could directly issue commands to arbitrary group of units. If you watch the replays, the APM/EPM is not too abusive. The main advantage is that it doesn't pay the cost of context-switching so it's very strong at multi-pronged attacks. It still didn't seem smart in the replays I saw on Youtube. Basically it's very good at developing mid-game timing attacks that humans tend to struggle with (not because of APM, but because it's easy to make mistakes when so much is going across the whole map).

I haven't played Starcraft II so I can't understand how well AlphaStar is playing, but if it has any sort of advantage other than its ability to develop a winning strategy, then it's very difficult to support any claims that it's winning because of its ability to develop a winning strategy.

It's a complicated matter because it's (claimed to be) the first system that's doing so well in Starcraft II so it's hard to believe it just has a mechanical advantage. But on the other hand, Google is in a position to throw a lot more resources on training their system than most others, so maybe it's just a combination of having a ton of compute coupled with a slight advantage in how you can control your units.

If that is the case we haven't really learned anything new from Google's achievement: we already know that a powerful computer can do some things faster than a human (e.g. arithmetic). We also know that there are things that humans can do a lot faster and better than computers (e.g. learn human language). The question is if Google's system is getting better at something that computers are known to be not very good at, in this case, strategy, I guess. That would be something new.

Personally I'm still in my default position which is skeptical.

The EPM is particularly worrying as AlphaStar can focus on multiple areas of the map at a single time.

A common strategy in GM is to attack on one front with your main army and drop a small amount of units into the back of their base. The opposition has to decide between spending their focus on the main battle or the dropped units.

AlphaStar doesn't have to worry about this - there's no gap between focusing on the front and the back at anything similar to a human.

Players usually use add hotkeys for camera positions to allow them to move around the map fairly fast. We can imagine a human player in this scenario responding by going

* Some actions to move army/setup * Use pre-existing camera hotkey to go back to base, view situation * Setup a new unit group * Move unit group to base to head off attack * Move workers to avoid economy loss * Go back to battle * Setup camera hotkey for battle * Flick between two battles as needed * Move workers back

If AlphaStar can view the two locations without that lag or without setting up new camera hotkeys for movement, there's a definite and sizeable advantage before we get into the amount of EPM involved. An action that is setting up a hotkey is significantly less valuable than one using a spell/moving a unit/attacking.

Just curious, not that DeepMind has to bring any return on investment anytime soon given how rich it’s Alphabet parent is and can easily continue to take out boatloads of money, but is there any foreseeable point of profitability for this company?

Or is this just more of a passion project from Sergey and/or Larry that they’re personally choosing to fund?

They have received worldwide news coverage for some of their work. Like Alpha Go. AI/ML aside, those achievements are huge in terms of marketing and PR.

Then, some R&D expenses can be tax deductible.

Just like... IBM?

Looks like R & D only so far. But they are in proximity to City of London. JP Morgan and Goldman Sachs are both staffing up their AI Research divisions. They all compete in the same talent pool. So I would not be surprised to see some sort of partnership with a major bank in the coming year.

Regarding multi-agent deep reinforcement learning. If anyone has caught any of the League of Legends play recently streaming on Twitch leading up to the Finals. Particularly Faker and STK. DeepMind certainly has it's work cut out for them. Even 99.9% expert level human play won't be enough ;)

Well, Google recently announced they were using BERT, an AI model, to improve search results. The developers of BERT are from Google but apparently not part of DeepMind; however, DeepMind has also published some things on language modeling, so... there's that, at least.

This project doesn't seem that expensive to run and it resulted in a Nature paper. Seems like a favorable ROI already even for a for-profit company (given Google's size and possible benefits in PR and recruiting).

Some Deepmind projects got integrated into Google projects in the past:

- WaveNet: https://cloud.google.com/text-to-speech/docs/wavenet - Data centre efficiency: https://deepmind.com/blog/article/deepmind-ai-reduces-google...

But yes, these are probably very small fraction of their research projects. Since Google declared to be an AI-first company, this is probably still a good investment to keep them on the bleeding edge.

Reply on news.ycombinator.com