Being able to ignore social pressures and just be yourself is definitely a benefit of privilege. So it's easy to see how many of the rarest skills and ingenious founder stories are examples of someone going against the grain to do something he or she cares about.
When people decry privilege, they seem to think that it's a bad thing, that we should all feel the insecurity, the doubt, and the drive to conform that most people feel.
I think the opposite is true, we should strive for all people to feel the empowerment of privilege and the freedom to pursue whatever moves them.
As a corollary, it's not a coincidence that the top schools admit the most privileged applicants. Those are the people whose privilege has allowed them to think about what is important to them... not just simply trying to be popular, avoiding getting beaten up by the kids who hate nerds, or avoiding ticking off an abusive parent, etc.
Privilege puts a person years ahead when it comes to moral clarity, purpose, curiosity, and self-actualization. Metaphorically, everyone else is in some sort of minimum security prison.