That's a very good question that we likely can't answer right now. "Woman" and "man" encompass a very wide range of bodies, with different averages, but whose distribution overlaps immensely on nearly every metric.

These gender categories are standins for some subset of differences that are actually causal, but we don't have the measurements and experiments settings to pull out what is actually causal from this.