Privilege goes in both directions: It's now become standard to diminish any white male's opinions on matter of diversity or gender or race, because of their identity, and give explicit power to anyone who is not a white male to criticize that group. Some say it's "leveling the playing ground" but it often just turns into a straight-on attack. It is now highly possible for someone in the "most privileged" group to be accused of X -ism and then be publicly attacked by outraged people, often with real job consequences.
By the way, I'm not white, but I don't think that should give my argument any more or less power. The post-modernist focus on identity politics and who is making the argument gives another channel for an ad hominem attack merely phrased in other words.
Edit: I can't reply anymore, but there seems to be a spectrum of offensiveness - and "lighten up" means moving the bar of acceptability in one direction, when people disagree about crass or potentially offensive humor - though of course the bar is different in a professional setting. In general I try to be respectful and conscious of potentially offensive things in a work-place. But among my friends (or among co-workers who have become friends), or at a comedy club, I'd imagine we develop a different threshold of offensiveness and be more open to crass humor. That's the "breaking the ice" effect Roizen mentions in her last example