There's definitely geometric differences between male and female faces. The most distinct difference is that the male chin tends to be larger, giving the face a more "square" shape, while female faces have smaller chins, giving the face a slight "almond" shape. And I'm sure there's more subtle differences as well.