Ah yes the old "give everyone in the world their own nuclear weapon" solution.
AI on the other hand has a wide range of extremely positive applications, some of which have such tremendous life-saving potential it's almost ridiculous. Many, or most of which, likely might never be achieved without AI.
The analogy is as ridiculous to me as calling personal computers in the 80's nuclear bombs because they could be used to hack and shut down the power grid. Yes, they could. And I'm sure some were scared of the prospect then, too.
A subtler aspect of this is the potential for cheap, tailored counselling. A few iterations and it will no doubt be possible to enhance the mental well-being of those without prior access, for comparatively little cost.
Those benefits again extend into every area, crime and poverty being rooted both in lack of education and lack of social and emotional support.
The social acceptance of chatting with an AI is important in this as it gets people over that mental hurdle. Localising chats so they are not subject to privacy concerns also advances benefits such as these.
There's positive benefits to be found everywhere with AI, but they won't be if we don't envisage, look for, and develop them. And they need to be found for balance, as it remains true there are many potential negatives.
This utopia you envision where we use AI for anything remotely good sure would be nice, but with the way the world and the people pushing for this AI especially work there just isn't a chance in hell that's how it's gonna end up going.
Any single not-for-profit social health organisation, of which there are a huge number, could use a tiny part of their funding to fine-tune an LLM on counselling resources and case transcripts. It'd cost little, and only need to be done once.
The major hurdle here, again, is education. Once such organisations realise how much more easily they can reach their goals using AI, they'll jump on it. The final obstacle is then social acceptance of AI assistance (growing now).
But the "extremely positive applications" to ChatGPT are, at the very least, the same positive applications of any other search engine.
I agree that comparing nukes to superhuman AGI is an imperfect analogy, because nukes don't have goals.
Good for clicks and getting an extremely manipulatable public coming back for more, I guess.
Historically, whenever we have created new technology that is amazing and impactful but that all of the positives/negatives are not fully understood, it's worked out fine. If we want to be scientific about it, that's our observable knowledge here.
Training a LLM is something random companies with some cash and servers are doing.
So the outcome of "pretty much only we have nukes" is an option. I don't think that's a realistic proposal for AI. Given that, is the outcome of "more trustworthy people stop, others continue" a good one?