Only asking for repo:public_repo should solve for the "I'm not giving you private repo access" concern. (I'm not, either! Nothing personal, but if that stuff was ready for prime time it wouldn't be private.)

The "fine-grained token" beta is what you really want to use if you can, because that does give single-repository access, which classic Github OAuth tokens do not. No idea how or if it's possible yet to use that type of token in your grant flow, but that's where you probably want to be looking.