What downsides?
I would add, in all seriousness, that it can make dealing with women a problem.
I think it's widely accepted that some men are apt do all kinds of foolish, prideful things in response to working with a woman they find attractive.
Some women are hardly different.
You can also be targeted by other men (of a different race for example) that feel threatened by you - or women too, the bitter type, in all seriousness.
To make it easier for your probable world view: being a very good looking tall black man for example, won't necessarily have everyone open up to you with smiles on their faces (especially if you don't necessarily want to assimilate into their 'culture'/social structure).
Being a very good looking asian (yes they exist, haha) guy myself - I would say I have experienced many instances in my life where white men have attempted to sabotage me.
People like things that make them feel/seem more powerful, and hate things that tend to do the opposite.
I would say, it's better to accept that these biases exist, so that you can take them into account when reasoning - rather than trying to delude yourself into thinking you are in entire control of all that is going on in your mind (and body).
the same thing happens to good looking white guys and good looking women (around other women) too.
you've never heard the term 'prettyboy'? it's used all the time to put down good looking men.
I still have not seen a credible source for that and basically think it's a PR stunt.