Facial structure, body shape, voice. Humans have sufficient sexual dimorphism and are attuned to the differences enough to make this distinction correctly almost all of the time.

This is how we can, in most cases, ascertain a person's sex regardless of how they're attired.

It's also why the other commenter's view of women being "clothes, hair and makeup" is so absurd. A change of clothes, a haircut, and wiping off makeup doesn't somehow change women to men, or remove the ability of others to recognize sex.

Also, even if a person manages to disguise their sex or impersonate the opposite sex, this doesn't change the reality of their sex. Just may obscure it from observation, for some observers.