At least the phrase gives you a point of attack if you disagree. You get to say "it isn't ad hominem because of reason X".

I find this far better than responding to arguments that deliberately attempt to obfuscate avenues of attack by couching everything in terms of "opinion" or "I have opinion X, but".