It's not even an issue of "mental resilience". Pair programming, like any other close human interaction, is strongly weighted toward the most aggressive or insensitive (or autistic) personality.

And in the tech world, it's very easy to encounter people more weighted toward the autistic end of the spectrum. Pair one of those with one of the HSP people, and you have a really shitty combination. The HSP will expend most of their energy appeasing and compensating for the needs of the autist. The autist will feel frustrated, and the HSP will feel fully exhuasted.