I've found this to be the right thing to do for several reasons.

1. Music does not improve my concentration in the long term. It will help for a few hours and then make it harder to focus after that.

2. Headphones make me less approachable at work, and I want to contribute to a positive team culture by being easy to chat to. If I need to buckle down I will work from home.

3. Headphones make it much more tempting to watch youtube, or listen to podcasts in the background. Without headphones I find it much easier to keep myself on track.

4. Most importantly, silence at work makes me appreciate music a whole lot more when I do listen to it, because it is no longer simply background noise, it feels worthy of attention. On top of that, I am more capable of listening to more demanding and more rewarding music.