In both cases, there are a lot of hidden variables. With voice, you miss out on non-acoustic things like beards, cheekbones, and other sorts of face-distinguishing features

With just a face, you miss things like the fundamental frequency (pitch) of the voice, dialect, and other linguistic variables.

In both cases, much is missing, and impossible to reconstruct beyond a stereotype.