It’s a perfectly valid argument. All invasions are generally bad, some worse than others. And, yes, some perpetrators of unjust invasions are punished worse than others.
The idea that you turn a conversation about fairness for victims into a conversation about fairness for bullies means you don’t actually care about victims.