Taking the example of games like chess and Go, where AlphaZero dominates, wouldn't we have to fear people misusing AI by setting up specific arenas and applying AI there long before AI misuses itself?
Drexler is more concerned about potential misuse by human actors – either illegal use by criminals and enemy militaries, or antisocial use to create things like an infinitely-addictive super-Facebook. ... Paul Christiano ... worries that AI services will be naturally better at satisfying objective criteria than at “making the world better” in some vague sense. Tasks like “maximize clicks to this site” or “maximize profits from this corporation” are objective criteria; tasks like “provide real value to users of this site instead of just clickbait” or “have this corporation act in a socially responsible way” are vague. That means AI may asymmetrically empower some of the worst tendencies in our society without giving a corresponding power increase to normal people just trying to live enjoyable lives.
The arenas of finance, business, journalism, and politics already exist, so we should watch for AIs trained to "win" those games.