Calling someone out for being racist or bigoted in some other should be acceptable, because that behavior is not acceptable. Preventing people from being called out on their bigotry is tolerating intolerance, and we shouldn't have any of that.
I do agree it is not a real rebuttal if the other side has a point unrelated to their bigotry.