I think you're underestimating the amount of latency that the human ear can detect. If the audio is as little as 10 ms out of sync, either way, someone will notice that.
Performers need to learn to ignore the rest and play their instrument while looking at the conductor.