To add on to what the other commenters have already said - this is especially true for marching bands where you are even further spread out and your distance to various sections changes over the course of the performance. You have to learn to (at least somewhat) disregard what your ears are telling you and focus solely on the drum majors for staying in time. Listening to the drumline to keep yourself on beat can be a very bad idea when they're half a football field away.

This effect is amplified even more for enclosed stadiums where there tends to be a large amount of echo, and can really be quite disorienting the first time you experience it.