Now, ad hominems aren't "good"... but if I work on it, I get lumped into the similar group. Just no.
"Just no", indeed. This is tremendously faulty thinking; damning someone and their entire endeavor, bypassing any and all rationality by your own admission,
not because they've committed some great evil,
not based on such a trivial concerns as the merits of the project,
...but because they have thoughts outside of that scope you don't particularly like.
Ad hominems are not bad out of some mystical universal constant, they're bad because they necessarily short circuit rationality by appealing to emotion, while saying nothing of actual substance. They are antirational and as such, as I put on my Sith Lord helmet, always absolutely bad.
Okay. He's a racist. And? Unless he's making a racial gating API[1], or otherwise taking evil actions, it doesn't even begin to have anything to do with the project. This problem that you mention, "getting lumped into a similar group", is entirely self perpetuating.
The statement "But he's a racist!" has zero professional value. The moment (the royal) you stops making such statements, the moment they cease to have value.
[1]: http://www.wired.co.uk/article/23andme-api-blocks-based-on-r...