I think the main issue is that the choice of the words "privilege" or "right", when used by enough people, has the power to set societal expectations. Why should a privilege ever be extended to more people, the people who have them should just consider themselves lucky they aren't taken away.

And I think there are plenty of things that are "privileges", but it's based on the idea that it is impossible to extend it to everyone. Being admitted to a top school is impossible to extend to everyone by the definition of a top school. Being believed in a court of law when it's just your word over someone else's, but being more trusted because you belong to a particular group is a privilege. Being let off with a warning at a traffic stop because you have the right face is a privilege. Living in the "good" part of town is a privilege.

Surviving a traffic stop without being shot by the cop should not be called a privilege. Being able to receive a competent education should not be called a privilege. These are things that can be extended to everyone and the words should reflect it, so that it's clear which things we should be fighting for, vs the things that are impossible for everyone to have.