Hmm. Suppose you have children. (I don't. Have been one, and have been around younger siblings.)
Now, you can do real violence to your children. Hit them until they're black and blue and sore for days. Or you can do trivial violence: smack your kid on the bottom with your hand and make it sting a little bit, for a fraction of an hour.
Maybe you could do this violence as an outlet for the rage, frustration, and anger in your life. Maybe you do this routinely. Maybe you're an alcoholic. Alternatively, perhaps you've found your children willfully doing something very wrong or incredibly irresponsible, substantially endangering someone's life or property.
These are substantially different applications of violence. We should use caution about talking like they're the same.
"To me, this seems like a glaringly immoral exception to a general rule: only use violence in defense of self or others."
One premise of punishing your children -- in ANY way -- is that you're defending your children's future. You hope afterwards that they will be a better person, capable of self-discipline, unlikely to engage in self-destructive behavior or violence against the world at large. And as they are not yet mature -- mentally, emotionally, physically, or in the eyes of the law -- and as they are your responsibility, substituting your judgement for theirs is a thing that you do, and you can in fact defend them from themselves, for a while at least. (It is a different situation with your wife, or at least it damn well ought to be.)
In that case, the question becomes whether the punishment is actually an effective and humane means to those ends -- physical or otherwise. After all, children can be traumatized with routine verbal abuse as well as physical.
"After all, pets can be controlled without hitting. Why can't kids?"
Pets have substantially different brain structures and the ability to learn from negative feedback of any sort is highly variable. My understanding is that dogs can learn from it, but do best if the feedback is immediate, otherwise they're at risk of just feeling randomly attacked, and they start getting pretty neurotic. But birds can just about never learn from negative feedback. And, of course, we have substantially different sets of expectations for humans than we do for pets.