Man beats machine at Go in human victory over AI
ft.com
ft.com
https://goattack.far.ai/human-evaluation
allowing one to replay all games. It looks like the strategy involves letting the computer completely surround a group with 2 eyes, and then depriving that surrounding group of 2 eyes (which the AI appears to be blind to).
Completely blind or just lacking depth perception?
I was somehow relating the word depth to search depth...
You are always exploiting the flaws of your opponent in these games. Winning is a matter of exploiting more than you have been exploited.
[1] https://arxiv.org/abs/2211.00241v2
[2] https://agagd.usgo.org/player/13822/ (6dan amateur strength is very strong, roughly 2300 Elo in chess)
Now they fixed the misconfiguration so their training could find actual weaknesses, and they found a very big actual weakness, that even weak amateur Go players like me can look at and say "oh my God AI what are you doing defend your group!". And it even generalizes to another popular Go AI, which is great!
This is a great result, very impressive, great work by the team.
"David Wu’s hypothesis is that information propagates through the neural network in a way analagous to going around the cycle, but it cannot tell when it has reached a point it has "seen" before. This leads to it counting each liberty multiple times and judging such groups to be very safe regardless of the actual situation."
(David Wu is author of KataGo)
I'm a little bit more skeptical now; it seems like what happened was that David explained some known pathologies of Go AIs (based on analysis by Go AI experts), and one of those pathologies (cyclic topology) was soft-coded into an agent and then run through the adversarial training process to harden it, producing the result in the paper. The arrow of causality runs backwards here - I would be far more impressed if the adversarial training process could help elucidate failure modes of AIs. This looks more like expert analysis laundered through adversarial ML woo...
Actually, at the time it was first trained, we were not aware of this pathology. We found out in later discussion that it was partially known in the computer go community. That is, cases where bots failed in cyclic ("loop") positions had been recorded. However, to our current knowledge, it was not known that it could be consistently targeted, whether by algorithm or by human, without very specific positions or specific sequences.
It's possible that would be a very slow, essentially "serial" process--adversaries don't really find a new trick until the AI can avoid the old, like with the pass attack in this paper. I'd also be curious whether training against adversaries makes the networks better or worse at general-purpose play or causes no change, but there's an argument that more-robust-but-slightly-worse-on-average is an overall improvement.
It also seems relevant to looking at weird behaviors of other models, like what we're seeing as people poke at LLMs more. Some of what we're seeing might simply be that they respond weirdly to some fraction of all prompts because they're not quite baked; also, we haven't defined their goals all that well yet. But automatic searches for problematic classes of input seem like a possible tool for keeping things more on the rails.
An engine which is tuned to beat Carlsen would demonstrate his specific weaknesses, for example specific issues with an opening he plays.
For his opponents, they already try to do this. “He plays this opening a lot” is a model of Carlsen, just limited to the first few moves of the game.
If you know the opponent won't spot a certain line, or will choose a certain sub optimal line, you would have an advantage by playing around that. Kind of like how you can play fools mate on a newbie.
I think the study confirms that weakness, and I am surprised that the tactics (to setup the circumstance for exploit) are easy to understand because those are actually being commonly taught to be strategic blunders to avoid. E.g. allows a group to be completely surround in early game is highly inefficient, even if the group survives with 2 eyes, it's only netting 2 points.
[1] https://blog.janestreet.com/deep-learning-the-hardest-go-pro...
AI, "Humans will take all our jobs!"
I assume this means they are supposed to be stronger than the best humans. If that is the case, this is a high level amateur defeating a stockfish-like engine.
Now the article mentions Leela Zero and that one is an open source version of Alpha Zero where they used a community effort to do the same as Alpha Go zero described in the paper.
From the status page here: https://zero.sjeng.org/
I see that this took multiple years but was declared finished in 2021. Status page says it played games in training are 21 million while Wikipedia says that for alpha go zero it was 5 million.
So you'd think it's better but actually it might just mean that they are not using they same techniques after all and so maybe worse.
So question for me still is whether this technique would actually be applicable to Alpha Go Zero.
But for me, this article is more a win for AI, because it means that they have found better techniques to find blind spots in models which can from now on be used in training to eventually get rid if those.
The fundamental issue is that they all do self-play, and those plays follow the same statistical distribution as a result. Weird games happen rarely, while the adversarial network forces a weird game every time.
On the other hand, I am not sure whether MuZero would be vulnerable. I would guess so, but its dynamics system may learn something extra about the state of the board.
Either way, it seems clear that future self-play training should include adversarial models, instead of pure self-play.
(They also all share another weakness, including MuZero, which is that their MCTS algorithm doesn’t allow the neural network to gain board information from their evaluation of possible futures; only win rate. Meanwhile a human sometimes realizes, while navigating a possible sequence of moves, that one particular early move could turn the tide.)
KataGo adds a few situations in training which didn't appear in self-play, "blind spots" that a shallow search depth fails to notice as good moves, so that it still tries to learn from positions it would not generate.
It is a little bit like learning to tie your shoelaces. Brains are not great at guessing the qualities of knots among all possible ties, but if you see it done once, you're set for life.
But the game is not solved, which is where MCTS comes in. It picks a move, and plays through it, selecting moves that are good but varied, until it learns more about which side would likely win (typically by expanding a leaf of the search tree). Each playthough is a visit. Each visit allows updating its estimate of the win probability for that move, and thus, changes the policy probabilities.
More visits typically anchor the policy estimates into reality further, yielding better play. In fact, this is how the policy network learns.
On a good consumer GPU (RTX 3060), I get 1.6K visits per second, so 100K takes about a minute. That makes a game last three to five hours typically.
But the paper emphasizes that even with 10M visits per move (>1h per move), KataGo loses 72% of games against this exploit[1].
[0]: https://arxiv.org/pdf/1902.04522.pdf
[1]: https://goattack.far.ai/pdfs/go_attack_paper.pdf#figure.capt...
Maybe it spends so much time over fitting for the hard cases it just doesn't have enough basic experience.
That's not what I said. What I said and what was my intended meaning was that humans can understand Go at a deeper level with a the aid of a computers. The computers don't understand go at all. They're software.
> long way away from an AI that truly understands the game
What do people mean you mean by this? An AI that can follow the process of human reasoning that mimics human understanding or an AI that actually reasons about any topic by having human agency and awareness?
Another point: I'm assuming The AI that lost can be fixed. Maybe very soon.
"After humanity spent thousands of years improving our tactics, computers tell us that humans are completely wrong... I would go as far as to say not a single human has touched the edge of the truth of Go." - Ke Jie, the number #1 Go player in the world at the time, speaking after his loss to an AI
To me it is like the invention of the microscope and how it showed there was a whole world of microbes we didn't know about, but no one says microscopes understand microbes.
Go AI have networks that map a board state to probability of winning. A perfect AI would correctly know whether or not any given board is a win for white or black. In general, better human understanding of the game directly allows you to better estimate if a board is a win or a loss.
Katago without tree search is plays at a professional level. I would say that this corresponds to at least professional level human intuition.
String comparison algorithms lack an objective "correct" ordering so it is hard to say what a correct sort should do. But a string comparison using utf8 has a super human (compared to any given person) understanding of alphabets because of the many people involved in ratifying the characters included in unicode.
: to have understanding : have the power of comprehension. : to achieve a grasp of the nature, significance, or explanation of something. 3. : to believe or infer something to be the case. (Miriam)
String comparison routines can sort strings like a super human, but they don't understand what they are doing. Neither does the Go AI.
I guess people are using 'understand' in a more casual sense, like my "rice cooker automatically cooks my rice, so it can be said to understand how to cook rice."
If you define understanding as what a human does, then of course only humans can understand. What's the point?
It seems that MCTS rollouts are a pretty good way of “understanding” a position. It is arguably one of the most unbiased ways. That fact alone suggests to me that it is at least very hard to judge. Maybe it is not interpretable by you or I, but I can’t say that rules out understanding.
I go back and forth on this. Machines aren’t introspective enough to have the kind of behavior most would call “understanding”. But then again, people aren’t either. And I don’t mean that in the most people are stupid but I’m very smart sense. I mean we quite literally don’t know why we have thoughts, we just do. We’re able to use words to provide justifications after the fact, but that’s not the same thing. The origin of any thought is a mystery to us.
Seems like it grasps the nature & significance of things to me!
It has a neural network trained on millions of games. Of course the model captures more than a sort routine which is non-parametric, in other words not trainable from data. The game of Go is complex enough that we can't code it up manually.
The fact that AI is implemented on matrix multiplication does not detract from it, humans are implemented in electro-chemical reactions. These reactions are all local, no single cell in our body has the big picture.
On the other hand, human understanding is also brittle - how many Go players still can't beat the AI even after this article has been posted? Blind spots/adversarial attacks exist in both humans and AI. It was AlphaGo that influenced the way Go is played at the top levels by introducing new strategies, techniques and moves that were previously considered unconventional or suboptimal by human players. AlphaGo found blind spots in our thinking first.
Now when professional Go players were defeated by AIs, it looked like these AIs mastered all those concepts. They excelled at playing a very balanced game, came up with very creative moves that had excellent aji, avoided bad aji. But now the adversary strategy, which can be explained in a few sentences to every Dan level amateur player, reveals that the AIs do not even have a good understanding of life and death. They fall for 'trick' moves that even I, a weak amateur (about 10k) player would have how to defend myself against. If you see some of the final boards, it almost looks the kind of outcomes you would see for players at my level.
As there are more generations of chess engines than go engines, it would be quite interesting to pull something similar off against them. My intuition is that it maybe works against Leela, as LeelaChess basically uses only neural networks (think alphago but for chess), whereas it should not work with Stockfish, as some parts of Stockfishs evaluation function are still adjusted by hand.
It's a bit hard to say for sure (since you can't pit them against each other, AlphaGo isn't available), but KataGo is very likely to be well stronger than AlphaGo ever was, unless maybe they contuinued it internally (unlikely).
If nothing else it's vastly more efficient to train and experiment with.
Are we taking this too hard?
I initially searched for a Major John Henry. :-)
There's also a cartoon (Chuck Jones?) about an axeman defeated by a Gyro Gearloose-type character (with a chainsaw) in tree felling.
But training on that particular sort of adversarial states should help against the human player which has learned the strategy, just like training on patch adversarial examples in vision helps against the same type of patches.
Of course if the adversarial policy is again allowed to find off-distribution states (by playing against the victim), it will certainly find ways to beat it, until the model is playing perfectly. (Emergent gradient obfuscation could also theoretically happen, but I don't know if it has been demonstrated to actually happen.)
We've apparently entered the stage where the deciding factor between who wins, man or machine, is just an arms race.
My understanding is that gwern above linked solid evidence in the paper for more search not being enough, as in, the model's evaluation NN is so way off target when searching, that realistic amounts of search don't help. Go seems to have many possible moves per position, so the search doesn't go very deep anyway.
Feel free to correct me if I'm wrong, it might be that I misremembered how AlphaGo-style systems work.
We're looking into doing our own adversarial training run to see if we can get to a point where the KataGo agent is robust. My personal suspicion given how difficult adversarial examples have been to eliminate in image classifiers and other ML systems is that although we'll be able to train particular vulnerabilities out of the system, there's still going to be a long tail of issues that can be automatically discovered and exploited.
In some sense it is more sensational than Lee Sedol beating AlphaGo, because it shows that Go is not yet beaten by AI and that this is only the true start of the battle between humans and AI with respect to Go.
For Go there is no such possible outcome, and deep reading (calculation) will always be in the machine's favor. The architecture of AlphaGo (policy and evaluation networks plus MCTS) is sufficient to recreate and surpass any kind of human-practical strategy. The era of human dominance in Go is simply over.
Of course, current AI programs will never be perfect either, so there may continue to be such adversarial attacks, but they will necessarily become harder to use (and quickly impossible for humans). AI go programs probably have nearly infinite possibility to continue marginally improving against each other, though.
This is an assertion without evidence. Leela Zero uses the architecture of AlphaGo and yet this guy just beat it with a human-practical strategy. I have no doubt that this can be fixed, but exactly how to fix it is not obvious and it may require more compute, and that's interesting.
When learning to play Go, you go to a number of steps, where sometimes you have to unlearn something you learned in a previous step. I am a weak amateur player (around 10k) and I only have a vague idea about aji (taste in Japanese) and no idea what good aji and bad aji is. I even don't master life and death. When AIs started to beat professional Go players, it seemed that they had a good understanding of aji, because they created good aji and avoided bad aji. They also played a very balanced game, overseeing the whole board, something that non-professional players often fail to do, and now the adversary strategy reveals that the AIs do not even have a good understanding of life and death and even fall for 'tricks' that I would have been able to counter.
It's the same reason why MOBA AIs get demolished by humans outside of extremely confined circumstances. The training space is much too large for the bots to handle every possible human strategy in games like MOBAs. So what happens is we just find weaknesses where they lacked training with non-standard strategy and we win. Reliably.
Yes, the bots do not lose lane 1 versus 1 with no outside ememy involvement. But once you lift that constraint, we do not lose to the bots as high level players. And it's not like there's only a single exploit, there's many.
In any case, I'm not expert in MOBA AI, but they are much more complicated than go just as go is to chess. So it will take longer, but it's unlikely the long-term outcome will be any different as long as Google etc keep working on it.
While I believe it's true these things might become practically unbeatable. I also am getting the feeling we've put god like faith in these systems when ultimately, we find they may have critical flaws and weaknesses like all to almost all other things.
Of course we're at the point where everything still seems like magic.
https://arstechnica.com/information-technology/2022/11/new-g...
Ah, they found an adversarial attack with another program. All neural nets suffer from adversarial attacks.
It's newsworthy because the AIs that were beaten seem superhuman at go. Unlike ChatGPT at writing code.