There is (typically) a difference other than pitch. Formants will be in slightly different locations [1]; the energy in the spectrum is balanced differently; there is more aspiration noise [2]. You can model the latter somewhat with this tool by moving the voice source box down.
I've been creating a speech synthesiser recently and it does seem like simple approaches produce a voice that is more passably 'male' than 'female', even accounting for the pitch. Aspiration added with white noise sounds better but doesn't quite get that breathy quality across. I think more sophisticated techniques may be needed.
[1]: https://www.researchgate.net/publication/220531604_Phonetic_...