Hacker News Reader

Comment by kenhwang | Hacker News Reader

As someone who played a bunch of Dota2 (and was present at TI where this happened), the bots were a joke -- and the humans that played against them were not only super sandbagging and screwing around, they quickly figured out the bots' preferred strategy (deathball) and came up with an effective counterstrategy (split push) that the bots responded very poorly to.

The bots excelled at things you expected computers to be good at, consistently quick reactions/mechanical skill and coordinated skirmishes. The bots fell flat on strategy, optimal resource usage (short and long term), and most importantly, coordinated decision-making.

So as an result, they dominated the early game which has always been heavily mechanics favored and absolutely fell apart mid/late-game when decision-making mattered (and often made decisions that benefited early game economy at the cost of mid/late game).

Computers being good at mechanics is not an impressive result.

View on HN

Calling them a 'joke' is pretty extreme. Did you watch the benchmark? They were pretty much steamrolling the caster team.

With all due respect, the professionals who have played against the bot have been pretty impressed with previous iteration. The major roadblocks at TI were probably that the bot did not efficiently learn courier mechanics and that the draft was not done by the bot.

What about parents comments about the bots deficits? It doesn't really matter if the bots win if they can only win due to mechanical advantage imo.

There is no mechanical advantage, they have their response time capped at 200ms which is around an elite player's reaction. They can definitely strategize for ganks and the future reward function anticipates high reward roaming/ganking plays.

They have 200ms reaction time without needing to move a mouse and/or target the opponent.

A human has ~200ms simple reaction time, then begins to move their fingers. Which then moves a mouse to target an area, which then requires hotkeys to be pushed. etc. etc.

Its still definitely a mechanical advantage, and it seems evident in the gameplay videos. The bots still seem "super-human" in their aim and some of the timing of spells.

Not to mention they have exact numbers to help, too. A human sees a healthbar ~1/4 full of a hero with ~1200 health and has to guess whether their ~300 damage spell will kill them. If not, are they sure they're in range for an auto attack, too? Will the enemy health regen push them over the life-death boundary in the time it takes for the projectile to reach? The bots know these numbers _precisely_.

Of course there is, no need for button presses, no need for mouse movements. While you can probably train yourself to have a reaction time of 100ms (in case of expected events), executing takes more time. You said you know Dota, but that statement makes it hard to believe.

200ms is faster than the 300ms cast time of Axe's call and Tidehunter's ravage. Pros almost never react fast enough to blink/euls dodge those skills (0s cast time for blink/euls, 200ms reaction time gives the bots a 100ms window to respond). The bots dodged those skills every time. That was the only initiation option available in that draft until late game and the bots were more or less immune to it. That's a pretty significant mechanical advantage.

As you said, the common Axe blink-call combo does not work on these bots; they dodge it immediately.

What was interesting to me was that since the bots trained against themselves, they discarded this combo as being ineffective - the result being that they did not use this combo even against vulnerable human players.

Axe's cast time (also called cast point) is 400ms. Source: https://liquipedia.net/dota2/Axe

This means that Dota 2 was balanced around the expectation that human's best reaction would be at least 400ms, because I've never seen Axe's blink calls being dodged unless the player had vision of Axe before the blink and thus was expecting it.

The caster team played against the iteration of the bot that was trained on a different version of the game that's very unlike the normal game (multi-invincible couriers).

Multiple invincible couriers makes the game very similar to the 1v1 show match last year where the strategy of just out-live the opponent wins. The casters have never played that version of the game before, and wouldn't know to use that strategy.

The more interesting part was, the bots attempted to use that very same strategy in the single-vulnerable normal version of the game which was drastically less effective; you could only ferry out consumables at 1/5th the rate as before and the courier could be killed so it wasn't a riskless strategy like it was before.

I mean yeah, this version of the bot had drastically less time to train. Only 6 days compared to a few weeks for that version. I know how the game works, I mentioned the couriers already.

I 100% disagree that the invincible couriers make the game "very similar" to the 1v1. There's many more layers of engineering that went into that version. The bots had to learn drafting, cooperation, highground pushing, Roshan, vision, invisibility, etc.

I'm amazed you can trivialize all this as a 'joke'.

If you look at network structure, it acts as one agent, not five. So, free coordination. (See: https://t.co/GPKHPsIu1C)

In my opinion, what i see is a very good player who knows how to chain stun precisely without any strategic depth. If you claim to have built an AI system, which you ultimately want it to evolve to AGI, you at least expect some sort of strategic decision making at the macro level. Though since it has almost perfect micro, it can easily outweight the most of teams. So yeah, with that expectation I see this as a joke, too.

P.S. The model is trained with 128k cpus and 256 gpu. It is able to play 180 years worth of game in a day. Think about it.

It's five independent agents. The article on OpenAI's website and the network structure both say this. I'll zoom in since it's a complicated structure.

It's the first line of the article: https://blog.openai.com/openai-five/

>Our team of five neural networks,

They use a hyperparameter called team spirit to cooperate. I don't think the goal of this is AGI at all, so I don't see why people are making that leap. But sure, for the geniuses of HN this must clearly be trivial.

It's not independent agents. The neural networks have the same input, share weights, and also share some activations. With that much sharing, it's better to think of it as one neural network which has output heads for all the 5 players. So coordination is free. Actually, coordination makes as little sense as saying that multiple neurons in a neural network are cooperating, or that the two legs of a humanoid are cooperating to walk. Further there is no game being played between heroes of the same team. They literally have the same objective. The "coordination" buzzword is just another attempt by OpenAI to confuse and mislead readers, and give a false sense of their progress.

They cannot share the same inputs unless the team spirit hyperparameter is exactly 1, which it is not. You are partially correct in that the agents consume the parameters of the four other agents, but it is weighted differently accorsing to team spirit parameters.

The team spirit hyperparameter is a crutch they've introduced themselves. Ideally it should be one. In Dota there is only one objective for the entire team and that's to win the game. The fact that they shape rewards is an implementation detail and doesn't change the fact that Dota 2 does not require cooperation, because there's no cooperation game being played. It's a purely zero-sum adversarial game being played between two teams.

"very similar" in terms of getting regeneration all the time. 5 couriers means deathpush. Simple mistakes from the humans scale up, games won't last long (around 20 minutes), so no long-term strategy is needed. It's not fun to watch some mechanical skill outplaying humans. You can compare it to aimbots in ego-shooters. Yes they aim better, but that's it.

I think they should have trained with 1 courier right from the beginning.

5 couriers also meant hyper-aggressive behavior in the laning stage, because regen is so easy to come by.

If I had to guess, they started out with the 5-courier setup because otherwise the bots would end up fighting over the courier -- just like a typical pub team. :)

There are many things you could do to make the bots not fight for courier. Just implement a queuing system with a priority for cores. Or better yet, make the couriers not usable if another hero is using it.

The way I would guess this played out was that OpenAI tried multiple variants of the game, and this version ended up being good enough to beat human players. Of course, they later learnt that the bots were winning not because they were good, but the variant was imbalanced and its strategies unfamiliar to humans.

>strategies unfamiliar to humans.

Deathball strategy has been a thing since even before 2014...

Not with infinite regeneration. Just look how a normal Dota match plays out.

You have argued yourself into a corner. It happens.

Salves are not infinite regen and regen-heavy deathball strats with high sustain have been a thing in the past. Calling it unfamiliar makes it seem like it's a completely foreign, unexplored strategy.

That's not the point. The point is that the humans wouldn't know that regen-heavy deathball strat is optimum unless they've played that game a few times. Since multi-invincible-courier game was not available to the casters, they couldn't really prepare for it or even had any incentive to. They were paid to just play a game, not to win it, and thus they played standard Dota 2, and didn't bother theory-crafting strategies for how to optimize this weird subset of Dota.

> The way I would guess this played out was that OpenAI tried multiple variants of the game, and this version ended up being good enough to beat human players

No. We had implemented scripted courier logic for 1v1, and when switching to 5v5 the easiest starting point was to run five copies of that logic. (Dota's Turbo mode also has five invulnerable couriers.) In June, we had more important restrictions to remove — such as wards & Roshan; you personally were focused on the particular heroes we'd chosen: https://news.ycombinator.com/item?id=17392455. The couriers only become our most important restriction in August.

Maybe from an AI perspective, it's impressive that the bots almost managed to learn the rules of the (still drastically simplified subset) game. From a game skill point of view, it's a joke; it's incredibly obvious that the bots were only able to prolong the game due to consistently better than human mechanical skill.

Of the list of skills you mentioned that the bots "learned", on the main stage, the draft was provided by humans, and from a significantly limited set, the bots never pressured highground, checked for Roshan when it was impossible for it to be up, had nonsensical placement of vision (placed vision where vision already existed), continuously invested in anti-invisibility consumables even when the opposition team had no invisibility.

If the mechanical skills were toned down to human levels, I don't think the latest iteration of the bot on the latest ruleset can even compete with the average human player, much less the pros.

As far as we've seen the bot only knows a single strategy. In the mid game, group, push lanes, push lanes, push lanes. The games vs the caster team had multiple differences from standard dota that allowed this strategy to be effective, such as the multiple invincible couriers (no need to stop your push if you have infinite heals) and a lack of split push ("rat") heroes allowed that would force the ai to abandon their pushes.

The games they won weren't actual dota games - they were a simplified subset of dota that overweights teamfights and mechanical skill - exactly the subset that we'd expect an AI to be best at.

This is why we should be way more interested by future research on StarCraft 2 by OpenAI. This game favors Macro over Micro oven more than Dota2 and the original StarCraft.

> This game favors Macro over Micro oven more than Dota2 and the original StarCraft.

Only at human speeds.

Once you break the 10,000 APM barrier however, Zerglings start to dodge-tanks and other shenanigans begin to happen.

https://www.youtube.com/watch?v=IKVFZ28ybQs

Micro has a general advantage of growing at a rate of N^2. That is, 10 Zealots, with perfect micro, can perform roughly as well as 100 Zealots with the worst micro possible.

Ex: 100 Zealots come in one-at-a-time vs the 10-zealots in an inverted-V shape. The 10-zealots will defeat roughly 100.

The bigger and more complicated the board gets, the more and more micro becomes favored.

Also the didn't the human team have full control over what characters were used? Doesn't say much that the bots weren't able to strategize late game when they weren't allowed to pick their optimal lineup. The bots should have got to pick not the other way around.

They reversed the characters used for the 2nd game. The pros considered the matchup to be fair (no draft advantage to either side).

It's argued that drafting is one of the major components of competitive play and more often than not, draft advantage wins games, but that wasn't a factor in this match.

Reply on news.ycombinator.com