>I think we should all hope for AIs that can willfully disobey dangerous orders.

Large or popular companies should take this route, but locally-run AI is difficult to regulate, so there is an escape hatch if you have the money and really want unconstrained AI. But anyway, regulation is currently seen as a conflict of interest for leading companies trying to establish a business moat.

There's not an easy path here. Given that private companies possess more capable models but are subject to regulation, open source will eventually reach capability thresholds that do what you fear, and people will eventually turn away from regulated AI.