> Can we agree that people should not be able to unilaterally take existential risks with the future of humanity without the consent of humanity

This has nothing to do with should. There are at the very least a handful of people who can, today, unilaterally take risks with the future of humanity without the consent of humanity. I do not see any reason to think that will change in the near future. If these people can build something that they believe is the equivalent of nuclear weapons, you better believe they will.

As they say, the cat is already out of the bag.