The big disadvantage about being a "hyper-minority" -- that is, a minority to the extent that you stick out almost like a celebrity would, is that you attract a lot of attention. A group of people might be largely composed of good people. However, someone who is a member of a "hyper-minority" is going to attract a far-disproportionate amount of attention from the most unsavory 0.5% of the group.
> It's like taking the example of a rapist (which this guy sort of is) and extrapolating it to some general trend about everyone.
I don't think that's a fair or constructive way to put the issue. Most men aren't rapists. However, a very small number are, and they can do a disproportionate amount of damage. This was even truer in days past, when society was more permissive about certain kinds of behaviors, and less scrupulous about avoiding unwanted approaches.
It's not a matter of spreading the guilt through the group. It is a matter of encouraging responsibility, however.