Somebody's going to build it. Not building it isn't an option. Would you rather the Chinese built it? The Russians? The Europeans? The Indians? A loose coalition of random hackers on the Internet, like the ones who build Linux?
What's wrong with what the Chinese have already built? It's not theoretical in that case. GLM, and Kimi are running on my infra right now, and that's something that I can never do with OpenAI's models.
As for the others, what would be wrong about European AI? I'm not European, but I find them to be a continent of fine people, with strong ethical values, there's no reason they can't take the lead on this piece of technology while the US deals with its rather more pertinent electoral and healthcare issues.
It turns out that is, in fact, an option. Even if someone else will do a bad thing, it does not make it acceptable for you to do said bad thing.
Comparing it to nuclear weapons is even more ironic from the only nation to ever use nuclear weapons. They are terrified of other countries treating them the way they've been treated, even though there is no evidence this is of even remote interest to anyone outside their hyperreality.
Firstly artificial superintelligence is a science fiction, it does not exist and it cannot be built using existing technology.
Second, plenty of stuff is possible to build, and is not built for one reason or another no matter how powerful. We never built our nukes in space for example. And there is exactly one reason why nukes in space was never built, and that reason is that we agreed not to.
By what authority do you claim this? Your school of thought has a really shitty track record of late.
How do you think Trump has made more profit during his years in the Oval Office than he did in 30+ years of business before that?