I don't get this logic. How does it matter if you are a sore thumb if you are not being discriminated? As someone who is also from a visible minority group, it never bothers me unless of course people change their behavior towards me because of the colour of my skin, which I have luckily never experienced.
Please don't feel that I am being condescending; I have come across this statement many times before and I can't understand it.