Why bother with SSO? If your family and closest friends use something like a wireguard client (iOS for example has a very good one that takes only a minute to configure permanently), the users simply switch a toggle and they are now on your private network and don’t need to SSO to anything (provided you have left everything open).
For a small home network the pros of that approach vastly exceed the cons.