If you're arguing that it's silly to place all our emphasis on a single group rather than trying to make society better by not giving preference to any group, that would be a potentially valid argument however, it's one that likely wouldn't survive outside a classroom setting.
History has shown us that the banding together of some groups (civil rights movement for blacks, women's suffrage for women) has worked to enact change.
I'd assume that a psychological influence towards change may be that it's easy to feel sympathetic to a group when you have a personal connection with one of its members. Having no clear members to connect with (in a world where no groups are given preference) would prevent people who are close minded from feeling sympathetic towards change. That is simply an assumption though. I tend to lean that way because it has been shown that facts don't win arguments(1). As such, I am doubtful that providing evidence to harassers about the ill effects of their harassment or evidence about the potential benefits of them stopping would have any effect on their behavior.
In addition, the problem of harassment, bigotry, and prejudice is one that is both complex and large in scale. It seems logical to break this larger problem into smaller batches (i.e. working on the most populous groups that are affected) in order to promote immediate change instead of waiting for a solution to completely solve the problem (which is unlikely to exist).
(1) http://bigthink.com/think-tank/the-backfire-effect-why-facts...