Whether someone is attractive or not, they're perfectly capable of not bothering someone who doesn't want to be bothered.
You're basically saying "if an unattractive man who doesn't respect boundaries is creepy, but also an attractive man that doesn't respect boundaries is creepy, then that means all men are creepy."