I don't think it should be uncomfortable to reject anyone, either. Being unable to reject inappropriate advances implies poor boundaries just as much as making the inappropriate advances in the first place.
The problem is, that the other person doesn't get to chose to participate in your game - regardless if he/she is comfortable with rejecting people or not.
If a person is walking around in the outside world, someone will sooner or later make some kind of large or small request of them. I can't see it as a terrible imposition for that person to be you.