"because it simply ruins the overall musical experience when an earpiece is in the wrong place"

I've long wondered: Is this true? It seems to me that the stereo field would simply be reversed, which apart from some hard-panning SFX on old Pink Floyd albums, I'm not sure that would affect the music in any tremendously meaningful way. I suppose instruments slightly panned to the left or right in the stereo field would be mirror imaged, but I'm still not certain if that really "ruins" the sound.