My main objection to this is just the added complexity. Instead of a single suid binary that reads a config file and calls exec(), now you have one binary that runs as root and listens on a UNIX socket, and another that talks to a UNIX socket; both of them have to do asymmetric crypto stuff.

It seems like the main argument against sudo/doas being presented is that you have a suid binary accessible to any user, and if there's a bug in it, an unauthorized user might be able to use it for privilege escalation. If that's really the main issue, then you can:

    chgrp wheel /usr/bin/sudo
    chmod o-rwx /usr/bin/sudo

Add any sudoers to the wheel group, and there you go: only users that can sudo are allowed to even read the bytes of the file off disk, let alone execute them. This essentially gives you the same access-related security as the sshd approach (the UNIX socket there is set up to be only accessible to users in wheel), with much much much less complexity.

And since the sshd approach doesn't allow you to restrict root access to only certain commands (like sudo does), even if there is a bug in sudo that allows a user to bypass the command restrictions, that still gives no more access than the sshd approach.

If you are worried about your system package manager messing up the permissions on /usr/bin/sudo, you can put something in cron to fix them up that runs every hour or whatever you're comfortable with. Or you can uninstall sudo entirely, and manually install it from source to some other location. Then you have to maintain and upgrade it, manually, of course, unfortunately.