You say you've never excluded women, but you are aggressively dismissing an attempt -- it seems, mostly from women -- at improving what they perceive as sexist undertones in the workplace.

If someone says "hey, we should work on addressing this sexism," and the response is, "be quiet, there isn't any sexism, you just hate fun," that seems pretty stinking sexist, doesn't it? Dismissing someone's concerns by saying they aren't real doesn't address the concerns, it just sweeps them under the rug because they aren't perceived as being important enough to give a shit about.

If some group has evolved office slang that includes saying "Quit being a ginger" to people who make dumb mistakes, and then that group hires some redheaded dude, wouldn't it be within his rights to say "hey guys, it's kind of weird to use 'ginger' as a euphemism for 'dumbass'"? Yeah, maybe he's "invading" and stepping on the group's ability to insult people at the expense of other groups, but it doesn't mean he's wrong.