It does take its toll, though.
I've been living in a big European city for more than ten years now. I can kinda forget about the noises surrounding me, but I always tend to feel less tense when I leave for the countryside. And when I get back, sure enough, I hear all the random noises again. Upstairs neighbor dragging a chair when sitting at the table. Other neighbor taking a piss. Neighbor from across, blasting his TV or his kid crying for whatever reason.
There is never a moment of respite in the city. You wake up, you got your neighbors doign their thing. On your way to the office / park / groceries, you got the traffic noise. At the office: the AC hum and useless office chat. On the way home, traffic again. Back home? Neighbors are back, too.