There is a fair chance that your core annoyance is something else.
That is, if you somehow got a perfectly sounds proofed workroom of your own, you would after a while realize that you're still irritable, find it difficult to concentrate etc.
Often, we don't want to face our real problems and assign discomfort to some external thing we have no power over. "If only X was gone, I'd be happy". But when X disappears little changes.
Since he's also permanently annoyed outside work, non work factors fit the data.
I commend you on the achievement of reaching a state of mental nirvana that allows you to concentrate equally in chaotic and quiet environments - very impressive. I would venture to say that the vast majority of the population will never reach this state.
Long story short, a workspace has to cater to multiple different people - shocking, I know.
People desiring quiet places or isolation to think isn't a particularly new phenomenon, so there's a good chance that this is the core problem.
That said there's a lot of things that can add stress and in isolation this one particular one might not be too bad. But then it becomes an optimization problem, there are many sources of stress that are hard or difficult to solve but the work environment is low hanging fruit and easy to solve so we focus on that.
This is why touring the office is an important part of a job interview, the places with a good environment will be happy to show you around. Fortunately some companies are so clueless they'll even advertise their modern open plan office, you don't even need to apply to those.
On the opposite note, I can perform extremely well in noisey environments if I am enjoying what I am doing - like I am able to just tune it out.
Circumstance and other stressors may be a main factor, or it could just be the noise. I imagine it really depends on the person.