Actually, i respectfully question that assertion. In my limited experience, groups of hackers have actually been much more on the homogenous side than random other groups of friends or freely-associating people (recent examples off the top of my head: a recent software developers meetup, or the makeup of the engineering department i work in – spoiler: it's overwhelmingly white, male, able-bodied and middle-class, straight). And IMHO, a lack of diversity certainly signals that something is going on. You might find the term "xenophobia" too strong, but exclusion is real and happens, _especially_, i would argue, among hackers. You needn't actively say something silly like "i don't like women joining my hackathon" for there to be a very palpable jock-nerd atmosphere that will turn off many folks that aren't white/middle-class/male. I think this might be an unpopular opinion here, but i believe it bears pointing out.
A while ago i read a very thought-provoking book entitled "Unlocking the Clubhouse" by Margolis and somebody else whose name eludes me. If you're not already convinced by the unapproachable ivory-tower side of STEM fields, that book may convince you.