Please see https://boardgames.stackexchange.com/questions/58127/ for reference. The first picture there shows a game supposedly "won by Black", due to a refusal to acknowledge that Black's stones are hopelessly dead everywhere except the top-right of the board. The "exploit" that the adversarial AI has found is, in effect, to convince KataGo to pass in this position, and then claim that White has no territory. It doesn't do this by claiming it could possibly make life with alternating play; it does so, in effect, by citing a ruleset that doesn't include the idea of removing dead stones (https://tromp.github.io/go.html) and expects everything to be played out (using area scoring) for as long as either player isn't satisfied.
Tromp comments: "As a practical shortcut, the following amendment allows dead stone removal" - but this isn't part of the formalization, and anyway the adversarial AI could just not agree, and it's up to KataGo to make pointless moves until it does. To my understanding, the formalization exists in large part because early Go programs often couldn't reliably tell when the position was fully settled (just like beginner players). It's also relevant on a theoretical level for some algorithms - which would like to know with certainty what the score is in any given position, but would theoretically have to already play Go perfectly in order to compute that.
(If you're interested in why so many rulesets exist, what kinds of strange situations would make the differences matter, etc., definitely check out the work of Robert Jasiek, a relatively strong amateur European player: https://home.snafu.de/jasiek/rules.html . Much of this was disregarded by the Go community at the time, because it's incredibly pedantic; but that's exactly what's necessary when it comes to rules disputes and computers.)
One of the authors of the paper posted on the Stack Exchange question and argued
> Now this does all feel rather contrived from a human perspective. But remember, KataGo was trained with this rule set, and configured to play with it. It doesn't know that the "human" rules of Go are any more important than Tromp-Taylor.
But I don't see anything to substantiate that claim. All sorts of Go bots are happy to play against humans in online implementations of the game, under a variety of human-oriented rulesets; and they pass in natural circumstances, and then the online implementation (sometimes using a different AI) proposes group status that is almost always correct (and matches the group status that the human player modeled in order to play that way). As far as I know, if a human player deliberately tries to claim the status is wrong, an AI will either hold its ground or request to resume play and demonstrate the status more clearly. In the position shown at the Stack Exchange link, even in territory scoring without pass stones, White could afford dozens of plays inside the territory (unfairly costing 1 point each) in order to make the White stones all pass-alive and deny any mathematical possibility of the Black stones reaching that status. (Sorry, there really isn't a way to explain that last sentence better without multiple pages of the background theory I linked and/or alluded to above.)