i've noticed that if coworkers tend exhibit a high level of emotion in their voices during their conversations, that's very distracting -- e.g., some people seem to just have a pattern of vocalizing stress or confusion in their tone of speech, either to each other, or on the phone etc.
it took me a long time to identify exactly what was bugging me (besides it obviously being noisy), but then it dawned on me, and now i pay close attention to it -- being able to cateogrize it, and then compartmentalize it away, helps a lot i think.
moving to an off-site coffee shop on occasion has made all the difference, despite being an equal or greater amount of noise.
still would prefer an office, obviously.