Gender, ie cultural context, does in fact play a role on how we speak and both the pitch and intonation of how we speak.
You ever hear of upspeak/uptalking? Vocal fry? It's not so simple as just sex driven characteristics. (Although testosterone does impact the development of vocal chords.)