A big reason why we even have someone at the top is so we have a finger to point when something goes wrong.
YOU ACCEPT AND AGREE THAT ANY USE OF OUTPUTS FROM OUR SERVICE IS AT YOUR SOLE RISK AND YOU WILL NOT RELY ON OUTPUT AS A SOLE SOURCE OF TRUTH OR FACTUAL INFORMATION, OR AS A SUBSTITUTE FOR PROFESSIONAL ADVICE.
Waymo, on the other hand, has liability insurance for every single one of their cars, and apparently they get a pretty good rate for their size because they have fewer accidents per car than the average person. The concept of fault is a bit different for them than for a single person. They would have to develop a pattern of systemic failures, rather than an isolated incident, before real questions of liability arise.
I'm from the UK, my best example of this is from 2001, when the Conservative Party started running campaign posters saying "You paid the tax so where are the trains?" despite being responsible for the privatisation of the trains before they lost power.
https://www.alamy.com/one-of-posters-from-the-conservative-p...
https://archives.bodleian.ox.ac.uk/repositories/2/archival_o...
[0] https://en.wikipedia.org/wiki/2016_South_Korean_political_sc...
They may choose to base their decisions upon the AI, but they will (should) still be ultimately accountable for choosing to implement the AI’s “guidance”.
AI can’t be elected mayor, so the person elected will be the one accountable for their actions regardless of who or how they are suggested.
There’s no need for any law about letting AI run the show because legally the AI is never running the show, the mayor is, regardless of whether they use AI, chicken bones, or a team of competent advisors to help them make decisions.
It’s no different than an elected official using their faith to guide decisions, they’re still accountable, not god.
That’s not deterministic. Also I’m not going to get into a debate on free will and determinism.
The more you try to work backwards from "I want to figure out how to make this illegal" you realize you get a lot of valid means of governing in the crossfire.