That's not a good argument. First, music is not speech. It doesn't require linguistic processing. Second, you can hear chords instead of individual voices, so polyphony can be perceived as shifting harmony, and it usually is.
It's not that it has to be like this, just that a mere theoretical argument cannot exclude it. And even then, musical perception has basically no relation to language processing.
I agree that language processing is different, but some similarities do exist.