Sure, there's average differences between groups. Between almost any two arbitrary groups you'll find significant differences.

The problem is that there's a very slippery slope between stating a difference, ascribing causality (the difference is due to biology, for instance), and justifying it as a moral imperative (we should treat men and women differently).

It's dangerous because if differences between groups are easy to pinpoint, they're much harder to explain. Even if you can get to a plausible causal mechanism, it doesn't say anything about how things should be anyway.