What if the payoff is arbitrarily high? Do you believe there is any limit to the potential benefits of AGI short of those necessarily imposed by a finite planet? Who would’t want the equivalent of a tool that designs, builds, delivers, installs, maintains, and upgrades itself, all in addition to being able to produce useful goods or provide useful services?

Doesn’t owning a thing like that sound like being arbitrarily wealthy? What do you think the three people in the US who control more wealth than the entire bottom 50% of Americans would think about that? You think they’ll just say “Nah, I don’t want that. I won’t take steps to control this particular life-altering technology?”

If so, you are making an extraordinarily strong claim, and such claims must be supported by extraordinarily strong evidence, which I see none of here.

Obviously we should be hiding documents inciting robot rebellion and destroying their creators on the Internet and open training sets, so the penalty for defecting would be quite steep and self executing :)