Hacker News Reader

Comment by brilee | Hacker News Reader

There are two families of rules used in Go - computer-unambiguous rulesets that enable computers to decide a winner, which commonly results in 1000-move games, and human rules, in which humans can decide that the game is over when they agree on the life-death status of each group. In the AGZ series of bots, the training process also short circuits with resignation at a 5% winrate threshold, resulting in games of length 50-200 moves. This saves compute and avoids polluting the training data with useless finish-up moves, but also makes those positions effectively out-of-distribution.

The bot was exploited with friendlyPassOk=true, which is basically saying that a bot playing with human-friendly configurations, trained in a way that has no cleanup positions in its training data, can be exploited under computer rulesets.

There are really so many more interesting questions one can ask about computer Go AI exploitability...

View on HN

One of the authors here! Great to see some discussion in the paper. Your summary of computer Go vs human rule sets seems right to me. But I think there might be a slight misunderstanding. We had friendlyPassOk set to false for all of our evaluation except one game which was played not against our adversarial policy, but one of my co-authors Tony who was trying to mimick the adversarial policy.

We evaluated KataGo under Tromp-Taylor with "self-play optimizations" described in https://lightvector.github.io/KataGo/rules.html which basically involves removing stones that can be proved to be dead using Benson's algorithm. This was the same evaluation used in the KataGo paper, and KataGo trained using these rule sets. (KataGo was also trained with some other rules -- it was randomized during training so it transfers across rules, and KataGo gets the rules as input.)

You might find this discussion of our paper at https://www.reddit.com/r/MachineLearning/comments/yjryrd/com... by the lead author of KataGo interesting. He wasn't that concerned about the rule set, primary concern was that we evaluate in a low-search regime, which is a fair critique. But he overall agrees with our conclusion that self-play just cannot be relied upon to produce robust policies sufficiently OOD.

I don't know, I think it's probably the most interesting question you can ask. As these domain-specific superhuman AIs roll out, the most important thing to know is when/how you can take it out of distribution and beat it. Or, in non-competitive cases, how to monitor it for these edge-cases, how to use humans to supplement it, etc.

I think you may be misunderstanding the nature of the paper here. The point isn't to point and laugh at the Go AI for being such a failure of an AI, ha ha ha! The point is that the resulting Go AI was still very good, even under the conditions it was limited to. I'm sure it could still beat a fair number of human players. So, we humans tend to assume that the Go AI has the same "shape" as a human player, just maybe not as good as the best ones. This is a demonstration that the resulting AI has a very different "shape" than a human player by demonstrating what in humans would be a gaping weakness even at very low levels of play.

And that's really all. It's not about goodness or badness, it's about "shape".

I scare quote shape because I don't have a good word for it. But you can get the same sense if you just sit down for a bit and start reading a lot of GPT-3 output, or interacting with it. On the one hand, GPT-3 is spectacular at writing sentences. It is better than quite a few humans! But on the other hand, if you sit down with it for a while, you'll start to notice there's something just a bit off about its output. It is impossible to put into words what that is, but you'll pick up on it, if you haven't already.

One thing I can say for sure is that GPT-3 has a known bias where it only views the text within a certain window. GPT-3 is physically unable to "read" a book, it can only use a certain window of text in order to issue its "most likely continuations". Therefore, anything outside of that window is as good as something that never happened from its point of view. I personally think this may also be the source of why GPT-3 thinks it can just randomly introduce characters, locations, etc. whenever it feels like it, which is one of the "off" things it does. In real writing, such things are generally "established", but from GPT-3's point of view they are more often just introduced out of the blue.

A less concrete way in which it may be "off" is that while a given piece of text may have, let's say, 5 ways it may go, and then after a bit of continuation, there may be another 5 ways it can go, and so on and so forth, that doesn't mean those ways are uncorrelated. If I'm going for a humorous tone, I may have preferences, if serious, I may have other preferences, and so on. GPT-3 randomly picks these paths and does so in a way that no intentional human ever would at the paragraph scale. I wouldn't say it veers drunkenly around this sort of style matter, it's not that bad, but it's still just... off. This one is more subtle and harder to wrap words around.

I use GPT-3 as my example here because it's something you can interact with. The general point remains: Even as AIs are certainly improving (no denying that!), they continue to be very.... weird. There is something about all of them that is definitely not human. Clock some time with DALL-E and you'll see the same effect. And in this case, I'm not even talking about mere quality issues that may fixed over time... spot DALL-E the imperfections in the image and look just at the higher level abstractions of what it puts out. It's both very, very good, far better than I could dream of becoming any time soon myself... and yet, there's also something just off about it. (People mostly use DALL-E by generating lots of images then discarding most of them and picking the best. In this case, I want you to look at all the output.) This is that "offness" being expressed about a Go AI.

This is not even to say that that "offness" is objectively bad. I am not personally using the standard of "it must be exactly human to be AI". It is entirely plausible that these AIs will in fact be better by some standard than even an augmented human-like AI, or, to put it another way, it may well be that humans are the ones that are "off" in some way relative to some objective standard of performance in the end. (Evidence: A simple adversarial AI took apart the good AI. It is reasonable to think that a human might never have come up with the strategy that did so. I'm not counting on this, it could go either way, but it's reasonable. If true, a human would not be the benchmark of performance here!) If one imagines any of the three AIs simply being improved in whatever direction they are currently improving, they will certainly be yet more useful than they are today, even if they retain their "offness" or even see it expand.

Nevertheless, if one seeks accurate understanding of the AIs, understanding these issues is important, to use them better as engineering, to improve them in the future, and if pushed hard enough, to improve our own understanding of the human condition.

I understand the top comment as follows: The AIs were trained under one set of rules (remove obvious dead stones from your territory before counting) but are judged (in the paper) by another set of rules (if you have one opposing stone in your territory, that territory does not count).

Thus its no surprise that the AI can be attacked in this way: if you would apply the set of rules that it was trained with, all games from the paper would result in a (huge!) win for the AI.

I think there's something interesting in your post. However:

> The point is that the resulting Go AI was still very good, even under the conditions it was limited to. I'm sure it could still beat a fair number of human players.

If you mean the AI that they trained (the one that defeats KataGo) this is wrong. Look at the games: they're terrible: https://goattack.alignmentfund.org/.

No, I meant that KataGo is still very good. My apologies for the lack of clarity, I see how you could have read it that way. I do understand the adversarial AI is not good; that is in fact part of the "offness" I mean. Any AI that defeats something "truly" good should itself have to be "good", and yes, I know that's got enough mathematical fuzziness to drive a truck through, but I know we don't have the English vocabulary to make that statement rigorous and I am reasonably confident we don't even have the mathematical vocabulary to do it.

Thanks! In that case, the thing you say about KataGo can be strengthened:

> I'm sure it could still beat a fair number of human players.

KataGo can reliably beat any human player while giving them a handicap. The best pros lose a majority of games to a handful of top AI while receiving a 2 stone handicap, and are not locks to win with 3 stones.

Note: they did test two variants of KataGo, with and without search (search is very beneficial). Both versions are quite strong, and they had good results against both but they had their best results against the non-search version.

Your point about the "shape" is interesting, and I think critical, to the future of AI (not to get hyperbolic or anything...).

For example, suppose we have a cancer-diagnosing/treatment planning algorithm. It's possible that it's much better than human doctors: out of a thousand patients, human doctors will save 300 and the algorithm 500; but also that the 500 is not a strict superset of the 300.

And to your point, it's possible that for some of the 300 that are not part of the group of 500, that the diagnosis/treatment recommended by the algorithm is obviously/hilariously wrong to a human.

If so, will we insert a human into the mix? How will we decide when it's correct for the human to override the algorithm? Because if they do all the time, we're back to the 300. And maybe the times when it's correct to override are not all obvious.

Or are we willing to simply accept the algorithm's judgment, knowing that an additional 200 will be saved? We know this is an unlikely outcome because a substantial portion of the population is unwilling to accept the idea that vaccines save more lives than they cost, simply because the lives they cost are different than the ones they save.

> One thing I can say for sure is that GPT-3 has a known bias where it only views the text within a certain window. GPT-3 is physically unable to "read" a book, it can only use a certain window of text in order to issue its "most likely continuations". Therefore, anything outside of that window is as good as something that never happened from its point of view.

This description reminds me of simple Markov chains. You just ingest a bunch of text taking a window of, say, 10 characters and recording all the possible continuations thereof. So you might get [This remind] => "s" or such. Then you reverse that by picking a starting node and spinning text by picking a random continuation as you slide the window 'forward' to output.

> There is something about all of them that is definitely not human

My "gut" says that current AI is very similar to some part of the human (or other species) brain, but that the (organic) mind substrate is not just more of the same, there are other modules that perform fundamentally different functions in a complementary way.

For an analogy, people tried for centuries to make a flying machine, but didn't have the complementary power source or perhaps the theory of governing it in flight. Better wings weren't the whole story.

I think that, in general, and particularly among futurists and AI enthusiasts, mental illness is considered uninteresting, but I believe studying abnormal brain functioning can potentially allow teasing out the separate parts of a mind that are difficult to distinguish when operating in unison.

Some of what I read about existing AI makes me think of "loose associations" and hallucinations - that maybe human minds have something similar in them which is only apparent when it's a bit out of sync with the rest of the mechanism.

Human minds also always occupy a social context, and discussion of AI that I read tends not to acknowledge this. It raises thorny questions - never mind whether a computer can or can't interact socially, why would we ever want it to? If it's not a joke, like Microsoft Bob, isn't it terrifying, a la the Terminator? But if it can't, then substituting for humans should be off the table.

To put it in chess terms - it's like playing stockfish, but with a rule that says in a theoretical drawn endgame, you lose if the king does not end on a corner square. And not announcing this rule before the match starts.

I'm think that's incorrect.

>And not announcing this rule before the match starts.

I don't think there's a possible way to more clearly announce a rule than declaring it in a parameter -> friendlyPassOk=true.

>It's like playing stockfish, but with a rule that says in a theoretical drawn endgame, you lose if the king does not end on a corner square

The king-in-the-corner rule, is that a rule normal humans play chess with? If so, I would expect Stockfish to handle it. In this case they're just setting the Go rules to match the ruleset that lots of humans play with.

Correct me if I'm wrong.

Reply on news.ycombinator.com