Hacker News Reader

Comment by baddox | Hacker News Reader

> The speed at which AlphaStar overtook previous AIs seems to me to be nearly unprecedented in AI research.

Is it not simply the case that, before AlphaStar, very little money and effort was being put into developing AIs for Starcraft 2?

View on HN

There's definitely some truth to that. But looking back on arXiv there are papers going back years, including from Harvard/UofT [1], Tencent [2], Facebook [3], etc. There has also been an SC1 tournament running since at least 2011 [4]

I'm sure that the AlphaStar effort absolutely dwarfed everything that came before in terms of investment (as I'm sure AlphaGo did for Go as well) but based on my reading I'd also bet that Starcraft has likely seen the most continued AI research effort of any imperfect information, real-time game over the last decade - I think that's also part of what made it appealing for DeepMind over Dota 2 or any of the other options in this class of game.

[1] https://arxiv.org/pdf/1909.02682.pdf [2] https://arxiv.org/pdf/1907.09467.pdf [3] https://arxiv.org/pdf/1906.12266.pdf [4] https://liquipedia.net/starcraft/SSCAIT

What makes this amazing isn't specific to StarCraft 2. AI in strategy games has been really lackluster. I can't think of a single example of a strategy game where an AI was competitive against experienced players due to strategy and tactics, rather than inhuman speed, accuracy or cheating. So it's not just about AI in StarCraft 2, but rather AI in essentially any (strategy) game. Now we have an example of an AI that can compete with real players.

There is a genuine advance here, but keep in mind that when an AI is developed by the game developers, they're not necessarily playing to win, but to make the AI fun to beat, and without using too much computer power, which would make the game slower. Also, due to commercial pressures it's tough to put a lot of effort into the AI for a game that's still being changed to make it more fun.

Even now, I wouldn't really expect to see most game developers start using the latest machine learning techniques to build their own AI's. Maybe the really successful games with competitive leagues might be interested?

>There is a genuine advance here, but keep in mind that when an AI is developed by the game developers, they're not necessarily playing to win, but to make the AI fun to beat, and without using too much computer power, which would make the game slower.

This point is being brought up a lot, but I don't really buy it. Yes, there have been instances where the AI being too good discouraged players from playing the game as much, but this almost always happens due to the AI having some inhuman advantage that a person could not really replicate. I think players do want the AI to pose a challenge - that's what a lot of casual PvP games are about. Humans are (were?) the only ones that could offer a fair match against another human.

I will absolutely grant you the computational power point though. However, as hardware advances, this cost will become more and more acceptable.

I don't expect to see widedspread adoption of this in commercial games within a decade, but I think that in 2 or 3 decades this will be the norm. In fact, I would bet that once we can make an AI that is good at a game, we can also make it weaker. A game could estimate a player's skill rating silently and then adjust the AI's strength to give the player a difficult/fun time.

Of course, it could just be that AlphaStar is able to play well against humans, because players treat it like a human. Maybe the AI still has standard game AI-like weaknesses that can be exploited if people know that they're playing against an AI. Eg some Diamond league player went mass ravens and kicked AlphaStar's ass. The AI would have to learn how to deal with stuff like this on the fly and I don't think we're there yet.

Well, yes, you need to ramp up the difficulty in a way that's fun.

The article describes "exploiter agents that focus on helping the main agent grow stronger" as key to their approach. That seems promising? Maybe it could be used to make bosses with specific strengths and weaknesses where part of the game is figuring out how to beat them.

I think part of the problem is that every videogame is a parlor trick in a sense. Unless you are a tournament player or someone who enjoys a really hard challenge, the bulk of the customers who buy the game just want to enjoy the illusion of a challenge. You don't really need a complex AI directing the hordes of enemies you want to defeat; you just want the illusion of battling enemies and emerging the victor, and how the game accomplishes this is not that relevant... unless the trick is so blatant that it breaks the suspension of disbelief, of course.

Why does this matter? Because the trick is often less effort than the real deal. The point of videogames is to entertain you, and the goal of game publishers is to turn a profit while doing so. Obviously there are some exceptions, and indie/free games might have different goals (we all know indie games that can be punishing, like Dwarf Fortress).

I remember with the original Left 4 Dead by Valve, they claimed there was an "AI Director" that modified the map according to how well you were doing, "adapting" to how you were playing. In practice it amounted to closing some sections of the map according to how easy you were killing zombies without dying. So this wasn't really AI but a glorified IF-THEN-ELSE trick, but what does it matter? It served its purpose. Do we really need something more complex (and expensive!) when killing zombies in L4D?

To extend your point, it is also clear from analytics in Steam & mobile gaming that games have massive amounts of player dropout. It is pretty rare, even in relatively short single player campaigns, to see 30% of people who bought the game finish it. So effort making the AI even better is always going to only be for an extremely small niche of players that play a game long enough to exhaust the limits of simple AI.

Just look at the achievements for Left for Dead. Only 40% of all players have got the achievement for "kill one of each Uncommon type" which is trivial to get if you play for a few hours.

Steam numbers are massively inflated by the tons of game purchases that come through sales, bundles, and giveaways and are never played. I have something like 600 games in my Steam library, and I would bet that the median playtime would be 0.0 hours...

Just today I saw something that looked interesting, and it was 85% off because of a Halloween sale. So I grabbed it for a couple bucks. Free time being such a rare commodity, there's a solid chance I never even install it... and then there's always Stellaris or Mount & Blade or Medieval Total War that I could happily play until the heat death of the universe, competing for my attention.

Isn't that just proving my point?

Most people who buy your game aren't going to invest even 10 hours in it or come close to maxing out the AI because they only picked it up because it was 85% off during a Steam sale.

Maybe you'll be able to figure out from Early Access numbers how popular your game is and whether you need to invest in super-awesome AI. But it seems like for most games it would be a waste of effort. More likely it would be retrofitted into an already popular game in a DLC or future patch or something.

>Maybe you'll be able to figure out from Early Access numbers how popular your game is and whether you need to invest in super-awesome AI. But it seems like for most games it would be a waste of effort.

Maybe the reason people quit your game is because of the AI. What made me stop playing Heroes of Might and Magic 6, Civilization 6, and Total War Three Kingdoms is the poor AI. It takes the fun out of the game, because at lower difficulties the AI puts up no challenge and at higher difficulties you're just exploiting the AI the same way over and over again to keep up.

You might think that that's fine, because you already got the money, but then you'd be looking at it from the perspective of singular games. But studios usually don't just release one game and then disappear. After my experience with HoMM 6, I had zero interest in HoMM 7. 2K Games could release Civilization 7 tomorrow, but without them showing that the game's AI is much better than in civ6, I probably wouldn't pick it up. Yes, the Civilization series will still have players, but it could easily just fizzle out like many other franchises and even genres.

You also can't add this in as a DLC, because people make up their mind about a game near the start. This only works if the game already has longevity, but at that point the game is already popular, so does a better AI at that point make a difference?

It doesn't matter how short or long the singleplayer campaign is. AI in games is so simple and rudimentary that you'll have it figured out in most games in less than an hour. You might not figure out the most effective ways to exploit it, but usually you'll figure out something. This alone can make the experience boring, especially when your game is heavily focused on playing against the AI.

While there are certainly plenty of parlor tricks and other illusions, I think in the best case, a game can contain a finely tuned learning experience, where by training you actually do get better at something. (Often something unimportant, but still fun.)

Throwing people into the deep end of the pool usually doesn't work so well. Scaling down the difficulty helps, but really what you want is challenges that teach you something. And you don't really need AI for that.

Last year I started learning to play accordion out of a method book. It has a bunch of songs ordered from easy to hard, and they're also chosen to teach certain skills. It reminded me quite a bit of good level design.

I agree with everything you said :)

>You don't really need a complex AI directing the hordes of enemies you want to defeat; you just want the illusion of battling enemies and emerging the victor, and how the game accomplishes this is not that relevant...

Yes, and the current game AIs are completely and utterly incompetent at it. AI in video games only provides a challenge when the mechanics of the game are challenging or the AI is cheating.

>Do we really need something more complex (and expensive!) when killing zombies in L4D?

Not in L4D, because you don't play the game to kill zombies. You play the game to play with other people. If L4D didn't have coop then the game would likely not have gained popularity.

You're right that players want a challenge that they can overcome, but current AI in video games does not offer that. The AI always ends up being so weak that play against the AI cannot be the focus of the game. The focus has to be in mechanical play, puzzles, story, atmosphere, multiplayer or something else. The only way AI keeps up is with inhuman mechanical play or cheating. Neither feels fun to play against, because the counter to it is to figure out how to exploit the AI, after which the game becomes more or less trivial.

> Yes, and the current game AIs are completely and utterly incompetent at it. AI in video games only provides a challenge when the mechanics of the game are challenging or the AI is cheating.

I wouldn't say "incompetent", but yes, the mechanics of the game must be challenging and/or (often) the AI must cheat. I'm saying this is enough most of the time; there's really no incentive to build a better AI for videogames because (my hypothesis) most gamers don't care. The illusion of challenge is enough, most of the time. The goal of videogames is to entertain, not to really challenge (again, with the exception of tournaments and competitive gamers, who are a niche within a niche).

You mentioned Heroes of Might & Magic in another comment. I played the heck out of HoMM2, some of (I think) HoMM3 and got bored with the rest. But I wasn't bored with the AI; I got bored because it was the same formula again and again. The genre itself soon felt like a tired formula. Note that the "AI" -- cheating or not -- of HoMM felt horribly difficult to me. I lost battles more often than I won; the only trick that reliably worked was to start the fight with a lot more troops than the computer enemy, and that's not much of a trick! Not a lot of places to learn effective tactics back then either, maybe the hellpit that was GameFAQs?

Hi Brian,

I really hope this will change, though. Skilled AIs could make so many games more interesting and better.

Many online games really suffer from bad multiplayer experiences. Players that drop in the middle of a game, ...

If Overwatch, Destiny, ... had really good AIs, that would make the game more attractive. If a player drops, they could replace it with similarly skilled AI. Instead of waiting for 12 players of the same skill-level, they could just play with some AIs.

They could even make it such, that humans win 70% of the time, and let the AIs just lose more often.

Yep, I'd like to see it too. If one popular game really did single-player with AI opponents well, maybe it would start a trend?

AI development probably needs to be made easier, somehow. Game developers have a lot to do.

I am not sure what you mean by 'strategy' here, don't Chess, Go, and No-Limit Hold'em fall under those categories?

They don't really count as the computer game genre "strategy games". Computer strategy games tend to have many orders of magnitude more complex rules and state than Chess, Go or poker, so the game-play is very different and making an AI for them is also very different.

The video game examples they give are what's called Real Time Strategy which is quite different from the games you described in which players explicitly take turns.

Chess and Go are already quite different from NLH, because they are purely about strategy. The best strategy in Chess either wins or leads to a draw if played by both sides every game. In NLH an optimal strategy just breaks even (ignoring the rake) against other optimal players and makes money on average against anybody else. But over even a few hundred hands you can't tell.

If you watch the alphastar games inhuman speed and accuracy are a big part Of why it is so successful

Is this AI better due to strategy and tactics? I am a low-level player but a fairly consistent spectacle, and in the videos I have seen I didn’t spot anything that looks like superior strategy or tactics. Just insanely good mechanics (which is still awesome!)

That depends on what you consider 'very little'. SC has had annual computer tournaments and research was published on it routinely. Few computer games see nearly as much money and effort put into developing AIs for them, yet the curve of progress was not exactly impressive.

Well, I don’t have any idea of absolute amounts of money, but I’d be more interested in relative amounts of money before and after DeepMind.

Starcraft is an incredibly complex game. 10^26 possible moves at any point (you can click/drag anywhere on the screen, pressing a keyboard button as you do so), imperfect information, real-time constraints, etc.

Yes it is, but you can prune the decision space significantly by discarding bogus choices, or more concretely by modelling higher level goals and then compiling them into actual moves.

That pales in comparison to starcraft at a higher resolution, which has even more points to click on screen! And even that is nothing compared to playing horseshoes in the physical world. There are an infinite number of moves at any given moment! And an infinite number of moments! Horseshoes is clearly the most complex game of all.

Plank would like to say something to you about lengths.

Which is why the most complex game of all is actually intergalactic horseshoes.

Starcraft is an incredibly complex game.

Even basic strategies will win if they’re done faster. APM (actions per minute) is a very significant factor into who is winning. Apparently they limited their AI player to 264 APM but that’s still incredibly high and done with machine level consistency. That’s almost 4.5 actions per second!! I know there are human level players at and probably above that level but that really allows for basic strategies to win out.

And the bot also can parse the entire screen in .03 seconds and then jump to a new area of the map. No human can monitor the entire map like the bot can.

But when the machines start competing with humans it should be fair?

Sometimes these critiques seem to be complaining that the ai is too intelligent. That’s sort of the point isn’t it.

If they want to model the constraints of humans, they probably need to create some “attention” system where the AI needs to choose where to invest its finite attention resources. For example, the AI could choose to focus more on the minimap, which would give them higher reaction times to move around the map, but would reduce their reaction times for things on the main screen.

A great example would be seeing the faint image of cloaked units. How does that work with the AI? Can the just phrase the current screen and instantly see any cloaked unit? I can imagine an attention system where more attention dedicated to part of the screen would increase the probability of noticing a cloaked unit there.

264 is not that high. People (if Serral is not a cyborg) can achieve even 1000 APM; and ~300 EPM average for a game.

Those aren't meaningful actions, though, are they? That's 17 actions a second.

Yes and no. There are many repeat actions in SC but at peak they can definitely reach that high. Microing units while simultaneously performing larger macro actions at the same time is a staple of any decent player and is really obvious at the higher levels. I personally play on an average of 100-120 APM for the game but regularly see 200+ in Platinum/Diamond.

> Even basic strategies will win if they’re done faster.

This isn't really true. Basic strategies done faster still lose miserably to humans, we can see this from the long history of SC Broodwar AI tournaments where they have a human play the best AI at the end (and always win).

Faster helps, better strategy helps more, maybe we can't say the AI is doing as well as the top humans at strategy since it's faster, but we can say that it's doing pretty good.

AI that ships with games (even the hardest difficulty) are made to be beatable. Of course pros beat them.

You misunderstand me, I'm not talking about the AI that shipped with broodwar. I'm talking about third party AIs made via bwapi competing against eachother in the various starcraft AI tournaments, such as

    AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE)
    IEEE Conference on Computational Intelligence and Games (CIG)
    Student StarCraft AI (SSCAI) Tournament
    BWAPI Bots Ladder

https://bwapi.github.io/

Ah, thank you for the clarification, I did indeed misunderstand.

>264 APM

Actually 792 APM. They limited their AI to 264 "things" (choose unit+ability+target or change view) per minute. Each "thing" can be counted as between 0-3 actions by starcraft. So it really has a peak APM limit of 792.

From the article:

>Agents were capped at a max of 22 agent actions per 5 seconds, where one agent action corresponds to a selection, an ability and a target unit or point, which counts as up to 3 actions towards the in-game APM counter. Moving the camera also counts as an agent action, despite not being counted towards APM.

But from replays it has an average APM of 248 at the highest, so nothing crazy going on. Plenty of pros have peak APMs during fights that are higher than 800.

Reply on news.ycombinator.com