But on the other hand I don't think the sensible solution is to leave everything unregulated! I am very glad that building regulations exist!
They might also work better if they say “you can't do it that way, which is known to be unsafe,” as opposed to “you must do it this way, which is the only safe thing.”
Note also that regulatory standards are not the only mechanism in the law to create safety. Liability law can be very effective at creating safety, by giving the right incentives to the right parties, but liability law doesn't tell anyone what to do—only what will happen to you if you cause harm.
A long those lines, governance power should not require being a celebrity either.
Applying this post to the topic of AI alignment, I'd like a democratic option for large entitlements of tax funds to be applied as Yudkowsky sees fit, if enough people vote to appoint him as AI Minister, and vote to give him governance teeth against big tech and the thousands of startups driving Moloch via the standard economic paradigm.
This isn't an option at all because our institutions don't adapt to the territory we often find ourselves in. We have a political alignment problem that exacerbates the technical alignment problems.
I would like to have something like the concept of senators, but not as location-based representatives, who are strictly assigned nothing more than to lead senate committees related to issues that we separately apportion funds and power to via direct democracy.