It's a small but very vocal branch of SJWs against a small and very vocal branch of counter SJWs, but both sides of it are in a meme war (using the scientific definition of meme: a mechanism to transfer ideas and culture) and the effects of their culture clashes are large enough for you and I to notice.
Also, the use of the term SJW might appear 'edgy' but its overuse has turned it into a marker of nothing but intellectual vapidity.
And the disadvantages faced by short people, fat people, non athletic people, poor people, ugly people, old people, uneducated people, etc etc etc.
My point being that everyone faces a different set of advantages and disadvantages that other people and classes either can't benefit from or prosper from.
The first is that "pointing out" is not where the story ends; like parent said, there is a small minority of people who use this "average man"(an amalgamation of features that doesn't actually exist in reality) to shame specific individuals(e.g. calling out novel authors for being "privileged" and thus shouldn't write characters with a different skin colour). This usage opens you up to an obvious reciprocal attack you would probably object to - one could point out that the average black person is much more likely to be arrested(https://ucr.fbi.gov/crime-in-the-u.s/2015/crime-in-the-u.s.-...) and use this to call specific black people "dangerous". In both cases, you are prejudging based on a stereotype.
The second objection is that once you start to delve into the "privileges", the label just doesn't seem to fit the actual things being described. Actually searching for "white male privileges" leads me to articles that are so easy to fight that I feel like I'd be addressing a strawman[0][1][2], so if possible, could you provide a list? My general contention is that the lists of "privileges" end up being a mix of: significant outcome benefits which have controversial correlation-not-causation interpretations(e.g. is blacks being arrested more "white privilege" or simply the amalgamated effect of high poverty, low education and latent racism?); things you would not really describe as "privileges" but simply different choices(e.g. men having longer careers due to spending less family time); things that are incredibly petty("I can if I wish arrange to be in the company of people of my race most of the time." from [0])
[0] - http://www.winnipeg.ca/clerks/boards/citizenequity/pdfs/whit...
[1] - http://everydayfeminism.com/2015/11/lessons-white-privilege-...
[2] - http://www.care2.com/causes/what-exactly-is-white-privilege-...
Well, your side has them, too. If you care about social issues, they are toxic SJWs. If you are a patriot, they are white supremacists. If you are a Muslim, they are Islamist terrorists.
Arguments are not soldiers [0]. Don't let a disagreement turn into a war. If you see someone support a cause you care about, but for the wrong reasons; and do so in a manner that alienates anyone who disagrees, you should call them out on it.
[0] http://lesswrong.com/lw/gz/policy_debates_should_not_appear_...