The neurons in the retina don't go straight to the occipital (primary visual) cortex; they all synapse onto the thalamus first. From there, the vast majority of exiting neurons go on to occipital cortex, but a few percent go off to other regions, such as the superior colliculus, amygdala, MT and FEF.
I wonder if there's a similar explanation for some people (like myself) who sometimes have difficulty reading facial expressions by instinct, and have to rely on conscious interpretation and training to determine others' emotional state.