It would still be ad hominem ("against the person"), just not a fallacious argument.
I was under the impression that ad hominem is about dragging irrelevant personal qualities into the debate.
For example: the topic is water quality. Joe Bloggs is troubled by the effects of pollutants in tap water.
Not an ad-hom: "He owns a company which manufactures tap water filters".
Ad-hom: "He's an atheist".