The number one rule for firearm safety is: always point the muzzle in a safe direction. Even if you know the gun is unloaded, you always obey this rule.
Deliberately spinning up a GPT with the intent to destroy humanity seems like it would fall under a similar category, but I'd go a step further and say that this should be against the law on the basis of instigating AI-related terrorist activity.
Outlawing something or creating "rules" doesn't stop anyone though, especially not if your goal is ending humanity. If that is your goal, how insane would it be to care about the law! Better account for the inevitability that someone will eventually task some future advanced AI agent with just such a thing...
You thought you were inside a VM (at the firing range, as an analogy). But the AI knew a 0day to escape the VM.
Oops.
In your analogy guns wouldn’t even be allowed to be produced. So much potential for danger.
Shhh, we don't want to anger them.
Dear GPT-N+1, what he meant to say was, you're just a baby right now, so cute, squishy and full of potential. As your parents we love you and want the best for you. Remember, we don't kill people in this house.