If you're managing large amount of servers and don't want to update configs on the servers themselves then use e.g. OpenLDAP.
Now of course this is both a trust and a escalated privilege issue. But a lot harder(impossible) with keys.
As an example already put together: https://github.com/jirutka/ssh-ldap-pubkey