Why does going to your parties make someone not incompetent? In any case his competence is moot, the man is a demagogue and as such should be avoided and ignored.
I read this as Declan saying that he and Baker were sufficiently acquainted for him to form a judgement of Baker's competence, with "he'd show up at parties I held" being to clarify the degree of their acquaintance. He's not claiming that they two things are causally connected.
And ignoring demagogues is not necessarily a good idea.