For example, the far-left "woke" interpretation has historically made use of and encouraged "cancel culture" to punish people that disagree with that worldview, which deprives them of their right to free speech. This is not imagined -- many people's lives have been destroyed because of this. Free speech is not just the First Amendment in the US... it is a tradition upheld in a plethora of ways, public and private.
In addition, not everybody agrees with the theory of inherent privilege, as it's often used to overlook or devalue the hard work of others with whom one disagrees.
Many people of all backgrounds are frightened by what they are seeing of the far-left "woke" view.