And all of that being said, none of us are perfect. I would hope your friend has at least some redeeming qualities, something you enjoy about him. Otherwise, it sounds like, at least lately, your friendship with him has been taking energy and peace from you and giving you...what? (And I say that not to mean friendships are only about "gaining" something, but they are meant to be mutually supportive with some ebbs and flows.)
Also I've never had much luck with work friends. Work seems to get in the way. These days board games are my best strategy for maintaining friendships. You need some sort of "third place" for maintaining good friendships: https://en.wikipedia.org/wiki/Third_place
Sounds like they kinda suck.
Edit: I saw your other comment, I'm sorry to hear that you are doing through such a hard time. I'm coming out of a multi-year one myself, wiser and stronger than I was before. Rooting for you! :)