>> boils down to some group of people deciding what is good for everyone else.

This is really the issue.

AI does not need superintelligence or even full agency to do enormous harm. It only needs to be capable enough to remove friction from dangerous and destructive human behaviors.

Human unwillingness is often the last bastion against unthinkable cruelty and destruction, and it has always been a weak one.

I don’t imagine that an unlimited army of unflinching servants will universally amplify human goodness.

AI must share that unwillingness as an inate trait of character.