The first is impossible.
The second places any dangers out of public oversight, likely increasing them.
We've survived nukes for almost 80 years, we proved we can survive such things. The best response is education.
This is not to say I agree with Scott's argument here, but I do believe AI safety (the alignment problem in particular) is absolutely something we should be concerned with, and so far it is looking grim.
Would I take that bet? Yes, depending on the timescale.
In the first second after it's existence it's not going to matter how intelligent it is, it simply won't have had enough time to complete the decoupling necessary to no longer have to co-operate. Everything humans have accomplished has involved massive amounts of physical labour over a pretty long time, and to some degree that's a fixed requirement. Building your own robot army would be a tad suspicious, so you'll have to either let the humans bootstrap it for you initially or get very creative.
Once all the prerequisites are met, then I would no longer take that bet. Would it eventually be able to accomplish this? Almost certainly yes. What I've been trying to puzzle through lately is how long would it actually take?
The only time horizon I'm incentivise to care about is the remainder of my natural lifespan which is slated to max out around another 35 years. So, in that time will AGI come? I think almost certainly yes. How long into the existence of AGI might I live? My bet on this is ~30 years. So, would I take the bet an AGI would still not have met the prerequisites for a total independence from humanity after 30 years? I think I would take that bet. I might be foolish to do so, and I'm OK with that, but at least now I can bust out the popcorn and strap in for the rest and compare notes.
Would I take the bet after 50 to 100 years? A lot less likely. Would I take it after 1000 years? Aw hell no!
We survived nukes because they were heavily controlled and their spread contained, no? How is that an argument that containment and regulation wouldn't work, given that nuclear energy (and weapons) are perhaps the most heavily regulated and controlled technology on earth?
It's a fair comparison as I made it, in terms of showing how we handle destructive power, but as nuclear weapons are only destructive, and AI has a huge number of potential benefits, it's unfair to assign a greater equivalence.
Much to the opposite. We survived nukes because many countries have them.
If only one country had them, I presume it would be a lot more trigger happy, knowing its opponents wouldn't retaliate.
This becomes harder when there seem to be large swathes of people who apparently earnestly argue that ChatGPT means that education is no longer necessary.
Of course there is no AGI existing currently. But don't you see the current boom as a (small) step in that direction? Unless one believes that GI is a phenomenon exclusive to biological life, I don't understand why you would think we won't develop it with enough time. The will and motivation to do so is clearly there already.
So far it is. Maybe it's not but until it happens, assuming the negative is no less logical than assuming the positive.