I agree with another comment here, you are not supposed to love pair programming. It's an attempt to put pressure on you. Just never do it, talk to your manager. If they are not happy with your decision, leave, as you are the one to suffer the consequences, not them.