That seems like the logic of a supervillain or sociopath. "When they did something bad, it's because they're a bad person that I'm allowed to hurt, but when I'm violent it's because I'm making the world a better place."
There may be times when that works, or it's the least bad option, but my instinct tells me that the sort of person that most deserves a punch is also the sort of person who is most likely to respond to that violence by doubling down and hurting more people (their victims often being those who can't defend themselves).