Conversely, one of the most distracting things for me is barely hearing people talking. My brain keeps trying to interpret the murmuring as words before it can reject it.
Same! My family loves calling each other on speakerphone. Asking however politely and considerately for them to use a headset or hold the phone is a mortal sin.
Coincidentally, when I moved out on my own, my ability to focus suddenly shot through the roof. Like putting on glasses for the first time when you're nearsighted.
Going back home I don't lose that sense of focus. Until the phone calls start again...
The thing about offices is that in my experience it's unstructured, so anything that can catch my ear might require my attention, and that takes a lot of focus away from the thing I'm actually trying to do. That is, the psychoacoustic form of the noise makes a difference, at least to me.
I grew up as an only child on a fairly pastoral 15 acres, so very different environment compared to most people who have siblings. As an adult, office noise bothers me a lot more than most, especially conversations.
The idea of being productive in a cafe / starbucks is almost absurd to me, like a Monty Python sketch. Although, I sort of feel the same about Open Office plans, which is less funny, especially in the Bay Area.