We are looking at a future where a security breach or misbehavior by one of a handful of companies could mass-compromise millions of businesses and critical infrastructure and possibly hundreds of millions to billions of devices. Even worse this permission is clandestine. It could be exercised against individual targets with no obvious audit trail, since auditing tends to also be delegated to the IAM provider (and nobody looks at local logs in most cases).
I guess we've been handing vendors a lot of power for a while with OS vendors that have "push" software update capability, but this is adding not only even more carte blanche permission to a small number of companies but extending it across systems running different OSes and even open source platforms like Linux and BSD.
Software update capability is also fairly coarse grained. Apple or Microsoft could push a compromised or malicious update, but it would be harder for them to specifically target a single user reliably and without being noticed. (Wait... why did I get a macOS update and none of my friends did?) SSO/IAM providers could easily do this. Imagine a subpoena that targets your SSO/IAM provider that gives the government (and maybe not even your government!) silent unlimited remote access to everything you have.
I can also imagine "cancellation" scenarios where a company doesn't like what you say so they dump your account and lock you out of all your infrastructure, requiring you to manually go around and "root" all your stuff. (If you think this would only ever be deployed against Nazis, study history a bit. Political winds shift.)
It's just a monstrous amount of power to give out, and I feel like people aren't thinking this through or maybe are not even aware of the power they are delegating.
I feel like people should at least understand what they are doing when they choose to delegate all their authentication to Google.