It’s mostly an issue that A2C isn’t designed for adversarial environments. It also doesn’t have any notion of hidden information, while other algorithms (eg CFR) explicitly handle this. There’s a well-known phenomena of cycling, where agent A will beat agent B which beats agent C which beats agent A; A2C can exhibit this. Think of rock/paper/scissors- AlwaysRock beats AlwaysScissors which beats AlwaysPaper. To avoid this, you typically need to do some sort of averaging.

The alphastar paper and blog post do a good job discussing these issues as they had similar problems. I’d say that’s a great starting point (and then following their references).

Blog post:

https://deepmind.com/blog/article/alphastar-mastering-real-t...