Nuclear weapons are not nebulous, vague threats of diffuse nature. They literally burn the living flesh right off the face of the earth and they do it dramatically. There is very little to argue about except "how" are we going to contain it, not "why".
In this case I truly don't know "why". What fundamental risks are there? Dramatic, loud, life-ending risks? I see the social issues and how this tech makes existing problems worse, but I don't see the new existential threat.
I find the focus on involving the government in regulating "large" models offputting. I don't find it hard to imagine good quality AI is possible with tiny - to us - models. I think we're just in the first lightbulbs phase of electricity. Which to me signals they are just in it to protect their temporary moat.