I found the opposite when I was in China, I don't think I had much of a social advantage although I didn't speak the language very well at the time.
Racism is certainly bad and should be discouraged. Somewhat controversially, apparently, so should heightism.
Or it is simply that you can physically intimidate people to get an upper hand in situations where such an upper-hand shouldn't be given... just because. How noble. What a useful and admirable trait to encourage.
There is that deterrent in place, that social constract. Sure it can be violated, but your argument is along a similar line of folks who advocate for open carry.
And if you are killed? There is a strong appeal to reducing the likelihood you will die.