if these models will eventually be as powerful as projected, these models should not be likened to guns, but to nukes. Which yes, if you manufacture nukes, we prosecute you
The most commonly accepted expected death toll of global nuclear war is around 50% (mostly from disease and starvation in the ensuing collapse of civilization). That's much better than the expected outcome of recursively self-improving AI (100% dead).