Privilege when used in this sense doesn't mean everything is great all the time, it means that in certain situations, some people have an inherent advantage because of something they had no hand in.

One such example is race, hence white privilege. There are other reasons for privilege as well.