I don't think that's what he's trying to say. He's probably referring to the fact that unless you're black/poc/whatever you're not allowed to have an opinion; it's the blacks/pocs/whatevers that are making him shut up, not the other way around.
To suggest that people should be absolutely silent and not contribute to the discourse that leads to policies that affect their lives is morally abhorrent. Ask them to weight their opinions, sure but telling them to shut up is absurd.
For example, you don't have to have been poor to contribute solve poverty.
You DO need a level of understanding and context, though. You don't have to be poor to contribute to solving poverty, but you do need to have experience with poor neighborhoods to know what the pertinent issues are and to get a more wholesome understanding of the predictament of the affected. I wouldn't expect anyone who's never stepped into a neighborhood with Section 8 housing to know the whole story enough to come up with any substantive solutions, and the same applies for racial issues, as well. If you don't know the stories of people who experience them and fail to properly evaluate their worldview, any opinions you have on the matter are based on faulty reasoning and can't truly be taken seriously.