It's the power that is scary, the power that you cannot control. It changes the balance completely. With AGI, humans are no longer in charge, therefore are irrelevant.
Most of the existing power structures that exist today have very little democratic control over them. The concentration of wealth has effectively zero democratic control, and wealthy people overwhelmingly control the entirety of how the planet operates. Even in the American election system, the primary elections of our two party structure are extremely undemocratic, and we're only allowed to viably vote on candidates that are chosen in their very undemocratic way. The problem compounds itself too when factoring in that voting decisions are overwhelmingly determined by media coverage, and media coverage is dictated entirely by billionaires.