Acknowledging the average differences between the sexes is neither colonialism nor sexism. But this on itself only tells you that there are some physical differences in size and muscular weight, and a couple months of the reproductive cycle where women will be physically impaired because of advanced pregnancy; it doesn't account for anything more.

Now, assuming that those minimum physical differences have an impact on social behaviour is sexist, because it fails to recognize the social reinforcement mechanisms that influence those decisions, from the structural separation of professional disciplines (i.e. the hard vs soft sciences divide, which dissipates when using multidisciplinary teams) to the social inertia of lower salaries of caretaking professions traditionally assigned to women.

BTW, assuming that these observations are produced from irrational anti-scientifism is the kind of blind spot of traditionalists that I was referring above.