I think the eye example is in fact different because light passing through the lense is inverted onto the retina as the root comment explains, there's no such inversion for perceiving other senses. I find the explanation intuitively appealing at first, but given connections are criss crossed for far simpler animals such as worms like you mention, I don't think the idea that vision is the primary driver of this phenomenon is correct.