I am not going to give you access to my private repos. Is that really necessary?
The "fine-grained token" beta is what you really want to use if you can, because that does give single-repository access, which classic Github OAuth tokens do not. No idea how or if it's possible yet to use that type of token in your grant flow, but that's where you probably want to be looking.
with regards to the public token thing it's a bit of the same complexity since I would need to know if the repository the user is using is public or private and then configure the OAuth scopes appropriately since I do want to have support for private repos (as that's how I use it currently)
It might get easier once fine-grained tokens leave beta. I don't know if OAuth support is on the roadmap for that, but it seems like a natural enough fit I'd be astonished if it weren't.