Yeah, it's a massive UX issue. The way to actually check if someone has a SAML session is to attempt to get their membership. If you get a 403, there isn't one. But good luck explaining to the user that they need to click "authorize" next to the organization in the OAuth flow. No way to send a hint that it may be required, and no way to do a step-up flow.
I did a full writeup here: https://notes.acuteaura.net/posts/github-enterprise-security...