There's a difference between these three statements:
1. Women and men are different - obviously true and too vague to be meaningful.
2. Women have breasts while men do not - obviously true and mostly (and hopefully) irrelevant to this discussion.
3. Women work fewer hours than men - A specific claim about women that may be used by some to justify their prejudices against women. I don't doubt that you made the claim without malice, but unfortunately these are the kinds of remarks that reinforce sexist stereotypes. For this reason, I wish people wouldn't make such claims unless there is watertight evidence backing these claims.