I think "a lot of people" is an over-statement, I don't think an intrinsically broken moral judgement made by a minority of people should be allowed to delegitimize the real issue, and I don't think the OP was talking about anything but the real, legitimate issue.

Privilege is nothing but an acknowledgement that some people have advantages in life that they didn't earn. There is no possible world in which it doesn't exist, and almost by definition its holders can't usually be blamed for holding it. But we can a) take steps as a society to decrease its impact in various ways, and b) see it as a reality-check on social-darwinist or meritocratic thinking.