I wanted to add that while in my PhD program I was studying this effect for a little while. In studies on ferrets scientists have discovered that the second auditory cortex, A2, tends to collect sonic motifs—things like a quickly rising pitch beginning near middle C, or any kind of warbling, or clear, constant tones at various pitches. Computationally, these motifs can be adapted to a particular corpus in order to improve detection and prediction rates while using a smaller bank of motifs. There's also lots of investigation into how these motif banks can improve robustness to distortions and noise.

As far as I know, it remains speculation that this effect occurs in humans, but it seems by your experiences (and my own learning Mandarin as a second language) plausible that the human brain itself "prunes" less useful sonic motifs in early life improving your ability to rapidly understand your primary language(s) even under shifts in accent or other distortions. This would likely come at the cost of literal inability to be attentive to motifs that haven't been stored. you would perceive them in A1, but they would have less resolution and perhaps semantic meaning (this is as far as I know quite wild speculation, mind).

I remember that it took almost an entire 6 mos of study before I really had a clue what tonality in Mandarin meant—I could hear it, but never perceive it as a linguistic phenomenon. After 6 months it appeared to almost overnight become something sensible to me at which point I had to relearn almost my entire vocabulary including the tonal information that I'd been unknowingly ignoring.