How would it make them worse off? If everybody agrees that AI is a threat to humanity, but also a possible edge over rivals, then the threat should be enough reason to get everybody to agree.

There are international agreements on tons of things that work the same way.