I've never particularly minded general office noises of the kind that can be cancelled out easily, but what gets me is when coworkers start to become stressed and physically vent this in a way that makes noise - particularly when I can hear a mouse click getting louder and more violent, echoing off every surface in the office. Somehow my brain interprets this as an attack (evoking a fight-or-flight response that others have mentioned here) even though I highly doubt it's conscious at all on their part.

The other type of noise that really affects my ability to concentrate is loud whistling that people will sometimes do as they walk down the halls, again reflecting off all the surfaces in the office, which my brain seems to involuntarily interpret as "Hey! Hey! Hey, hey look! Hey!" and constantly resets my focus as though someone had just tapped me on the shoulder every few seconds.

When I'm not trying to concentrate hard on something I tend not to be bothered by even these noises - but when working it can feel almost like being physically harassed. I suppose this is a form of misophonia, but it is extremely context-dependent and at other times I enjoy the diversity of sounds (including the whistling).