The harassment women get is much, much worse than men.
Me and my wife are both computer scientists, and have both given talks at many conferences. My wife and our female friends consistently get much more, and much worse, harassment than me and our male friends. There is no comparison, and over time I've seen it grind several women down to the point where they just give up (fortunately not my wife yet).
I think everyone should help to reduce the often hostile attitude in IT and FOSS in particular, and a fairly obvious part of that is to stop treating women so badly. Just treating them like everyone else would be a fine start.