It's not clear to me how your argument would work for GPT-4 when it's clearly not reasonable for nukes.
Dito for the sewage/water system or other critical infrastructure.
Not saying OpenAI needs to be elected or not, just expanding on what (I think) they meant.
If we accept that the public having access to GPT-4 has the same level of risk as the public having access to nukes would than I'd argue that we should treat GPT-4 the same way as nukes and restrict access to only the military. I don't think that's the case here though and that since the risks are very different, we should be fine with not treating them the same.
At least with ai you can cut the power, for now anyway.
With something that can be so trivially copied as a LLM that isn't possible.
So in this scenario, one could argue that ensuring equitable distribution of this potentially dangerous technology at least levels the playing field.
North Korea is dirt poor and they managed to get nukes. Most countries could do the same.
Then, you could say the exact same thing you're saying now... but in that case, nukes-slash-nuclear-energy still shouldn't be distributed to everyone.
Even nukes-slash-anti-nuke-shields shouldn't be distributed to everyone, unless you're absolutely sure the shields will scale up at least as fast as the nukes.
Problem is, if it only takes one person to end the world using AI in a malevolent fashion, then I think human nature there is unfortunately something that can be relied upon.
In order to prevent that scenario, the solution is likely to be more complicated than the problem. That represents a fundamental issue, in my view: it's much easier to destroy the world with AI than to save it.
To use your own example: currently there's far more nukes than there are systems capable of neutralizing nukes, and the reason for that owes to the complexities inherent to defensive technology; it's vastly harder.
I fear AI may be not much different in that regard.