Have you talked properly to the colleague in question? It seems like you've bottled up all this frustration for far too long, making this infected, which in turn makes it harder to talk about it.

Even if it's hard, just tell him how you feel. Take it easy (don't start by listing all his faults) and stay objective - as in "When you do ... I feel ...". Be honest and say that you're not sure if you want to stay, and that the situation needs to be resolved so that you can have a healthy working relationship again.