If the allegations that he kept a shag matters in his office are true, i don't think you can actually find a defensible position for this.
I don't like the outrage culture and twitter witchuhnts either but this seems to be more than digging up a bad joke someone said 10 years ago.