With most information hidden, the game Stratego had stumped AI until now
arstechnica.com
arstechnica.com
Too bad I never played against an AI before they cracked it.
I think this is a problem with many deeper games. I remember going over to other people's houses and they would pull out obscure-board-game-xyz. The rest of the night for me was trying to figure out how to play, while others were 30 steps ahead of me.
I think you need something like chess club - stratego club - where people have all gotten to the nuance level and you can play with people of appropriate level.
At one point in my life I got in literally thousands of games of Dominion, and I was definitely still not as good as some of the people I discussed the game with online, but in person I could barely find anyone to play with, ever. (In fact, of the people I knew who were familiar with it, few even seemed to think it was any good as a game.)
I don't think Dominion is a good game. It's good if you maybe think cubing is good. And it can be a lot of fun!
But as a game? It's not good. If you have no idea what you are doing and are totally unfamiliar, you can even win pretty easily with some luck. But you won't understand why you won, or rather why others lost.
There are many, many games where everyone can read the rules together and be, more or less, on even footing. Dominion can sometimes be that way, but it can pretty easily be extremely opposite of that.
Interesting, this is a _feature_ of poker games like Texas Hold'em. If the losing players didn't go on convincing winning streaks, they'd stop playing. And those winning streaks can convince them they don't need to learn anything else about the game in order to be a winning player.
Have you ever gambled over a game of Dominion? I wouldn't.
Have you ever played Hold'em without gambling? I wouldn't either.
My experience as well unfortunately. No matter how many books on chess I read, no matter how many games I played against Battle Chess to try to get better, he always won.
But about having nobody to play because you crush everyone - as long as you want to play to enjoy and not just always go full-on try-hard mode - just stop crushing everyone and you'll have people to play with.
Let people win every third game. If you're good enough (at it), they won't notice. They'll have a good time and you will. In Tekken it works to have them win some rounds (extra points if you let them have two out of three), but still win the match, tho maybe only if you're not just 2 ppl playing.
That's a rule established from rat behavior observation study, you can hear about it in every third jbp lecture.
Ever since implementing it, I never end up with nobody to play with. Just curb your wanting to always win and focus on maximizing the amount everyone enjoys playing in the long run.
Even a game where there's always a winner and a loser doesn't have to be a zero sum game.
Sure, I'll go back and tell my 12 year old self to do that.
But, actually, I did do that, to some degree. As I mentioned, I would drop hints and talk about my strategies in games, usually after the game, so he could be more competitive and understand how I was thinking about things. That also has a cost on the fun for both parties, though. Smart people can recognize when you're going easy on them. Nobody normal likes being pandered to.
What was the secret of your success?
It's been a loooong time, and I don't recall all the details. But it revolved around doing probing attacks to determine where the ranks were in the enemy formation, and then having "channels" in my side to move up a soldier that outranked by 1 a targeted attack.
Color me surprised that it would be difficult to write a program to play it.
One thing Stratego did was implement the "fog of war". My Empire game took inspiration from Stratego and Risk.
And, I'm also working on a game that kind of pulls on that same kind of upfront planning, where how you place your pieces to start is as important as how you move them; a settling and strategy game that's heavy on clever defense and using terrain and the interplay of pieces.
Imo, this is the critical piece and what makes the AI work at all.
With hidden information games, the best move depends on information you don’t have. So a move could be good or bad, it just depends on something that’s impossible to know.
You’d like to search ahead, meaning “if I do this they will do that” but that’s impossible since you don’t even know what the opponent can do because you don’t know their hidden state.
If the possible hidden states are randomly distributed, you are screwed. It’s just like rock paper scissors: there’s no best move if your opponent is unpredictable.
However if you can quickly learn to predict their moves, it becomes possible to make informed decisions about what to do.
The practical difference is how many rules a game has and how easy it is to implement the engine. Implementing a chess bot is relatively easy because the amount of state tracking required to set up a simulation is basically nothing (I think just whether the king has made a move yet or not). That makes it easier to implement than something with a lot of signals that need to be recorded. Something like DoTA or Starcraft takes serious engineering effort.
> There isn't any reason to think AI have more or less trouble with hidden information games.
How about the fact that a child can beat the best rock paper scissors player in the world in a game, but no human can beat the best chess engine? Same thing with poker, a novice could get lucky and win a hand against the best poker player.
I thought this was slightly less crank-coded than trying to prove the Riemann Hypothesis, but maybe these days you just ask Claude to do that and it tells you there's a counterexample at 1 + πi that no one ever noticed before.
(Of course, tomorrow Google might announce that it has solved it.)
IMO it also needs to use a real mouse before I think it's a true comparison, even a casual player would have a massive advantage if they could issue selections and unit commands via query.
I personally don't see why vision is important, if anything I'd frame vision as useful to humans, rather than being the baseline.
Pluto uses Ghosts to counter Dragoons. Any SC:BW player should know how stupidly impossible this is for humans.
For the non-players out there. Ghosts have an ability called Lockdown that can stop mechanical units (like Dragoons) from attacking or moving for long periods of time.
The problem: it requires you to click on a Ghost, then click on a Dragoon.
No human in the whole history of SC:BW would seriously do this. At best, Ghosts in practice are used vs capital ships (like Carriers or Battlecruisers), because in smaller numbers humans are fast enough to click and perform these feats. (Carriers and Battlecruisers are very expensive, very very powerful, units. So they are weak to specialized attacks like this as you cannot build many of them due to their incredible cost in resources).
But dragoons? This is the mass production unit of Protoss. This is so far outside the scope of human-level APM that the games Pluto plays are useless. Lockdown simply isn't feasible to be used vs what amounts to the standard / stock infantry unit of Protoss. There's simply too many opponent dragoons to even click on.
-------
I dunno what kind of "APM rules" we need to give SC:BW bots to allow them to have a fair fight vs humans.
But the rules we have in place today do NOT represent a fair fight at all. Computers have stupid amounts of speed and precision far in excess of a human player.
clickbait title and the bot isn't at "top players struggle to beat it" level - but it's no longer a casual walk in the park like the default ai is, so the progress is massive
Just 16 GPUs, and a few thousand dollars?
What about “researchers from Carnegie Mellon, MIT, New York University, and Stanford University” this wasn’t just anyone.
so the contrast here is the budget available, not the quality of the talent, if we accept the premise that DeepMind and the universities have approximately similar level of talent.
So clearly it’s due to talent as well as advances, of course.
Those new additions can invalidate the whole training data by a single new "card" that changes completely the dynamics and would be easy for a player to understand and incorporate but not for an algorithm (perhaps with enough compute to re-train it regularly it could) - that along with the decision trees being orders of magnitude deeper, wider and with more conditionalities than go, chess or stratego - even through the same turn with the same cards available and same table state - would probably pose much harder problems for a compute bound algo.
This doesn't follow. You're basically proposing that new combo decks be added all the time, and it's far simpler for an agent to scan the new cards for potential interactions with the thousands of other cards in circulation than for a human to remember all of them.
Your analogy is akin to saying that all you have to do is keep landing new code all the time, and since the agents weren't trained on the code they won't be able to identify and respond to security vulnerabilities in it as fast as humans, which hasn't turned out to be correct
But on MtG in particular that never really applies in full due to drawing new cards. You can play perfectly and still lose due to sheer randomness of draws.
The latent space I'm not sure how it translates to a game playing bot, but I would imagine that it would open it up to fail in the same ways a human fails.
On the game I'm designing it could do that (calculate all possibilities up to X depth, for all possible scrolls and table states) but it would be extremely expensive to do so (not a very good argument if compute power keeps increasing), but more than that, in contrast to something like chess, there can be many more paths and decision points where a bad decision turns into a loss, so if it assumes that the best play is X at some point, a sequence that it discarded due to not being the most probable can exist and the bot can never be sure, so if it makes a decision that plays into a "trap" he can't undo to a favourable position. While in Chess it's much clearer what is possible from a given state, it's unambiguous and the rules are fairly limited.
In stratego you have a 10x10 board game, a very clear objective and at most 40 pieces (with repeated pieces and simple mechanics amongst them), while in MtG and similar games a single piece (card) can have probably hundreds of different interactions depending on everything else going (and everything else hidden), at many points of decision. In stratego it also seems that for humans at least, most moves are "inconsequential", as it probably plays more at the psychological/bluff level. Maybe a human player that was given the same budget for training could spend a month training against bots might fare better as the strategies might be then better understood (by the article it's mentioned that the agent recovered from bad positions, so it seems that it was mostly human error, as the human was playing better up to that point).
While on MtG or Asummon, although there can be inconsequential moves (they don't matter given the context/stage of the game), every move carries with it a possibility of being consequential in unpredictable ways. Anyway, there should be ways of training models with just a rule abiding client for these games, without codifying all rules, that they can just keep playing to figure out the interactions, so if that theory is true then it should be possible to create an unbeatable bot - I'm just not sure it is without infinite time/compute and less so if the "meta" keeps changing rendering possible training inconsequential regularly.
I didn't look for prior work on this, but my estimate is that it's probably within 2-3 orders of magnitude of additional training compared to a static game. (Still a lot!)
When the Dota 2 bot was made, they retrained the bot only partially when new patches came in, so it was definitely cheaper to adapt.
There is nothing that, in principle, makes MTG different from poker or bridge, and we have superhuman engines for both.
We don't have superhuman play for bridge.
Poker and bridge are quite different from each other in terms of solving them. Among other things, the hidden information space in poker (at least, in hold'em) is far smaller than in bridge (or Stratego, for that matter, as discussed in the linked paper). This makes hold'em solvable using CFR, an algorithm which essentially optimizes play by considering all the possible holdings than the opponent might have and their best strategy with each one. Even going from two to four hidden cards per player (Omaha) requires a slightly different approach although you can still use CFR as the basis for the search algorithm.
Bridge has 13 hidden cards per player which makes CFR basically impossible to apply, at least in any obvious way--just way too many states. Similarly you see it's not used at all in this Stratego paper.
There is - metagame. There is no universal optimal strategy in a trading card game, because what is optimal depends on what decks and strategies other people are playing.
I'm sure you could train a neural network to play a specific deck within a specific metagame of a specific card game, but you would probably have to keep re-training it when there are new decks/combos/releases/rotations/banlists/metagame shifts.
In my own game you don't have shuffle/draw randomness but the pool of options is statistically tending to infinite (if I would have 500 or 1000 scrolls designed and MtG depending on the format has that depth) when compared to something like chess, or this game. On the other hand in my own game you have to account for much more depth on the possible options your opponent has.
Ex: you don't really care if the opponent plays Giant Growth or Chastise. The effect is that the opponent is playing a combat trick, and combat has moved from attackers favor into defenders favor.
To defeat an instant speed combat trick requires a combat trick of your own, or a generic counter spell of some kind. Some have interactions (ex: Doom Blade beats Giant Growth but not Chastise), but the overall gist is that opponents can do things after combat is declared. You only need to keep track of how many combat tricks you think the opponent has.
---------
Other situations are card advantage (ex: 2 for 1. If the opponent spends 1 cards to defeat only 2 cards of yours). The traditional card for this is Mindrot, but well placed counterspell can turn a combat trick into. 2-for-1 reversal.
You don't necessarily keep track of how your opponent makes 2-for-1 opportunities. You just have vague gists of them.
---------
Good spells have huge applicability. Doom blade or Murder is high because killing opponent creatures at instant speed handles the vast majority of creature buffed combat tricks, and also serves as a way to stop enemy combos and other such tricks.
In contrast, chastise is very niche. If the opponent were playing like Swords to Plowshares (powerful white instant speed removal), it's pretty much always better than chastise.
If the opponent plays chastise instead, you take that as a win because you know they could have had a deck of better cards. But for whatever reason decided to play with weaker cards...
But even then (not saying I'm right) I think the depth of choices, effects and so on, on a format like modern, or legacy, would be very difficult for an AI to top against pros. If you add draft into the mix it gets worse for the AI in my view too.
Because a good play in most situations can easily be a bad play under others. That doesn't happen in chess for instance, given enough decision depth to the algos to see the future game. In my own game I think those situations can occur much easier due to you always having your full deck available. Also, in MtG it's easy to get into table states that are either ahead/behind and then you kinda just have to protect your position (like with denial decks). Then you have the effects that you might remove a creature threat (graveyard) but then that enabling a combo you weren't expecting that needs a creature on the grave, or enabling delve cards or whatever have you. It's much less clear cut for a probabilistic model to make the optimal play at every single interaction. So the more you train the model on all the variations and possible follow ups, the more you dilute its certainty isn't it? In chess, or this game, or RTS such as starcraft, that doesn't really happen in my view.
No human can possibly keep up with all the possibilities or combinations that are accounted for.
Games of incomplete information have been IMO soft-solved as of.... Maybe 5 years ago? As in, stronger than any human can possibly reach (ie: massive GB-sized matricies accounting for all information iterated over millions of iterations of "he thinks that I think that he thinks that I think that....")
It's not a true Nash Equalibrium, which remains outside of the realm of even computers to compute. But a computer can always reach a closer / better estimate of any Nash Equalibrium, which covers all games of incomplete information.
--------
For Poker, it turns out that a few types of bet sizes (3x pot, 1.5x pot, pot, half pot, quarter pot) covered enough betting patterns to reach superhuman.
And frankly, MtG is simpler than the bluffing game in Poker. Like MtG has bluffs but it's no where close to Pokers level.
There's no crazy deep game for Red Deck Wins vs Control. The game basically plays itself out (Red tries to win before Control comes online. Control tries to stall before Red Deck Wins). There are some games with complex board states but they're largely a game of bluffing + card counting (opponent holds 4 cards, two of which were since the start of game and 2 were top decked in the last two turns. He at best has only planned for 2 responses or got lucky with the other two newest cards. Do I have a play that beats two cards yet?)
Only in the most general form they are games with cards and hidden information with a state space that some form of tree search can theoretically play out.
The difference is the size of the search space. In MTG the search space is unimaginably huge. It would make Go's search space look like a spec of hydrogen in the middle of the universe.
It would require completely different techniques to produce a computer good at MtG than one that is good at bridge.
I like Stratego a lot. Next time, I will try this and put three bombs in the other corner to keep the opponent guessing and wasting an expected 4 extra moves.
I speculate a big part of the game is moving your power pieces (1, 2, 3) to places where they can actually attack the opponent safely. Any other placement of the flag and bombs creates a bottleneck for left-right movement on your side.
[1] https://www.hasbro.com/common/instruct/Stratego.PDF "When an attack is made, the attacker is the only player who has to declare the number of his or her piece. The defender does not reveal the number of his or her piece, but resolves the attack by removing whatever piece has a lower number from the gameboard. Players keep their own captured pieces. Exception: when a Scout attacks, the defender must reveal the number of his or her piece.
1. https://boardgamegeek.com/boardgame/3513/electronic-stratego (We generally banned the use of the 'probing' feature)
Bridge is played as a pair vs pair game, with North/South and East/West being the two pairs and seated around the table in these compass directions. A bridge hand consists of two phases: there is first an auction phase, where players go around the table bidding on contracts (agreeing to take a certain number of tricks with a certain trump suit) until a final contract is decided. Then there is the cardplay phase, where the player who won the auction is the declarer, their partner is the dummy, and the other pair are defenders. The dummy's hand is placed face up on the the table and the declarer controls which cards are played from dummy, so the cardplay phase is effectively played by only three players now, with each of the three knowing one common hand (dummy) and one private hand (their own) and not knowing the other two hands.
In both the auction and (for the defense) the cardplay phases, it is important for players to exchange some information about their hand to their partner. However, any information you exchange about your own hands also helps your opponents. You might naturally conclude that you want to come up with some secret scheme to exchange information which your opponents don't know (and it is even possible to exchange encrypted information which your opponents can't know--if the defense is known to hold a certain card, but declarer doesn't know in which hand it is, the defense could say that a signal means one thing if the card is in one defender's hand, but means a different thing if it's in the other defender's hand).
But it turns out that this ends up being very uninteresting to play, so instead, when playing bridge, there is an important rule: all of your partnership agreements must be public. If a certain bid that I make promises that I have at least 5 spades in my hand, it is the opponents' right to know that this is our agreement. You must be able to explain the information which your action provides, and you must be able to use the information that the opponents give you themselves.
This poses several problems for self-play reinforcement learning. First, a naive self-play approach will produce agreements that cannot be explained to a human. What really needs to happen is that your partner, when determining what hands you might have as part of search, must not do so simply by sampling its own system (ie by asking what it itself would have done with hand X or hand Y). The information and possibilities really need to be mediated by some kind of intermediate, rules-based description, which can be provided to the opponents as well.
You also need to be able to encode and ingest the opponents' agreements, and to use this information to inform your own decisions. And you need, in particular, to be able to handle a wide variety of agreements from your opponents; it's not enough to force them to play the same system as you.
You must also account for deceit. If, for example, I have a bid which promises that I have at least 2 cards in every suit, it's perfectly legal for me to lie and make this bid when I only have 1 card in some suit--as long as my partner is in the dark about this just as much as the opponents. So if you make this bid, and your machine opponents assume there is a 0% probability of you having lied about your hand, it is possible that they will make gross errors by not accounting for this possibility (for example, they may be in a position where all of their actions are equivalent if you told the truth, but where one action is clearly better if you didn't--a human player will naturally take this action, but a robot may just select an action randomly).
It's an interesting game and a very interesting AI challenge.
How does AI, in a game as complex as bridge, manage to deal with a human partner, or even an AI partner? Seems like an answer we could find out.
> The information and possibilities really need to be mediated by some kind of intermediate, rules-based description, which can be provided to the opponents as well.
This is standardized at tournaments (I'm sure you know that.)
Still a few humans writing on the internet... at least for now ;) I'm sure the green username doesn't help either, I just switched to a new account a few days ago.
>This is standardized at tournaments (I'm sure you know that.)
Right, in human play we have convention cards (pieces of paper that are essentially big forms for specifying common agreements), although these don't cover every situation.
What I was getting at was more along the lines of some kind of general schema that would allow fully describing any individual bid. But then there's also a problem where such a schema inherently limits the creativity available in constructing a bidding system, if all the bids must fit into what is expressible by this schema.
My personal approach would be to try to decompose it into two subproblems: learning a set of agreements, and optimizing results given a fixed set of agreements. Then you could try to solve the former problem using the results from the latter one. But even playing well under a fixed set of agreements isn't so easy to solve.
I am particularly disappointed that it has influenced how people play the game.
The joy comes from the journey and the experience.
Look at competitive chess and Go and how they have fundamentally been transformed. It's not better and now the box is opened, it can't be closed.
For people like me, competitive chess killed chess long before Deep Blue. It only feels fair that competitive chess players now feel like I did :-)
I had a math professor who played competitive chess in his youth. He told me he realized the demands for competitive chess were such that he couldn't really dedicate himself to math (or any other discipline) at the same time. So one day, he gave up chess - and refused to play it for the rest of his life.
> The joy comes from the journey and the experience.
> Look at competitive chess and Go and how they have fundamentally been transformed.
Go is better since AlphaGo. Tools are better, it's easier to learn from your games, we're better at it. The AI makes sick fucking moves and we get to see.
The journey is still there, the experience is still there.
Chess I doubt is worse off either, but I don't know chess that well.