Ironically, by dismissing your input because of your race or gender, they are engaging in the exact behavior they are purportedly against. That kind of juvenile hypocrisy should be avoided by everyone, but sadly, is not.
Beyond that, you don't have to be poor, or female, or black, or... etc., to understand empathy. In a world where empathy didn't exist, maybe they would have a point. But humans can and do empathize, and can bring in elements from that divergent background to improve the overall situation. But only if the discussion isn't cut off with, "STFU, what do you know!?"
This is the worst kind of rhetoric. Probably all groups who were ever oppressed suffered a little bit more than just people criticizing them. I guess your empathy did not go far enough to realize that?
take this example - while a lot of str8 men may now be worried someone will falsely accuse them of being sexist or homophobes, I literally do not know a single gay person (I'm gay, I know a few gay people) that did not experience physical or verbal violence at least once.