It's a shame, because I think it used to mean exactly the opposite, having a privilege was an honor. Is the phrase "it was a privilege to have known you" still being used?
I have intimate knowledge with writers, specifically offspring of well known writers who do their best to hide this fact, yet will still use their parents agent. And yes, it is unfair for them to believe they are on equal footing with any other writer.
That doesn't mean they don't try hard, but they do have benefits built into their life that others don't.
If you want to trailblaze without being perceived as someone of privilege getting there then you should take paths that align with that, and not ones that don't.
I used to think about this, "only people with relations can make it". When I got older I realized that if you care about some subject, you will probably end up making relevant connections on the way.
And I also think this way of thinking about privilege is self-defeating. If you don't have "privilege X", find another way.
Some writers now got rich without ever having an agent, self-publishing.
I'm sure people will find some other alleged privilege they had, which allowed them to do that. That's not the point.
If you are dead poor and have no connections whatsoever, you have the "privilege" to write authentically about being poor.