The thing about punching down is that, because of the power differential, you don't hear back when you've gone too far. You may even get the victim smiling and laughing along with you, for their own safety.

But people remember, and talk among themselves, and your name becomes known, and maybe one day there's a huge backlash. And the backlash looks disproportionate, but should really be considered in proportion to the decades of incidents it's in response to.

(You can see this happening in real time with statues of racists, especially in the UK)