What do you think you're gaining by walking up to a white homeless man freezing on the street and explaining to him how much better his life is because of his "privilege"?

If you think that was an unfair summary, well, a) it was, and b) it also quite clearly explains why "privilege" is a losing argument if you're actually interested in increasing the amount of justice in this world. Privilege sounds automatically unfair and feels unfair and people's minds will slam shut. Do you really want to expand the reach of your ideas, as opposed to just virtue-signal to fellow activists? Then drop "privilege."