The problem is, that the other person doesn't get to chose to participate in your game - regardless if he/she is comfortable with rejecting people or not.
If a person is walking around in the outside world, someone will sooner or later make some kind of large or small request of them. I can't see it as a terrible imposition for that person to be you.