I am a happy user of https://tailscale.com which I think solves the same problem
I'm not super averse to Google SSO but I would not tie my critical infrastructure to it in any way.
Another dealbreaker for me is hosted configuration and access management. I want to be the only one managing it.
Regardless, it would have been nice if you could use your own identity provider.
A fair bit of the client code is also on the repo. Things like role based access control that require a backing store on tailscale side aren't open.