I like to step back and just say that it's not something that has to be fixed if a cadre of people appear to "collectively decide" based on individual decisions. It sucks that if you are in the cadre and have to fight the perception that you should(n't) be doing something. But I think (as I suspect you do as well) that saying that your individual decision should be different simply because you share an identifying trait with others is broken.
Ideas that men or women are some collective that needs to be pushed one way or another are difficult. You can generalize based on the apparent preferences of the group, but then judging an individual based on those preferences is where it starts to get troublesome.