I agree. For instance, (and it's been a while, so perhaps my reading is bad), Arendt makes a distinction between the justified use of violence, in which we suppose that violence is wrong but can see no other course of action, and legitimate violence, where we understand violence simply as our right as people with power.
A justified violent action is future oriented, hoping that the future will vindicate what will be a past action but knowing that at the moment, the action is "wrong" until it becomes redeemed through historical context.
By contrast, a legitimate violent action is past oriented, and understood as simply "right", because the power carrying it out has been granted legitimate state power. Here, we can see how people in the US who favor military action often see military action as the "right" of the US to protect its interests, coupled with the legitimacy of US power as a force for the promotion of human rights (not a view I agree with, by the way).
This might sound like a strange and useless distinction, but consider the difference between a police officer shooting someone because there is no other action, and doing even though it may be wrong, and on the other hand, a police officer tasing a "non-compliant" suspect because he believes he legitimately has that power and won't be harmed even if, in the future, people decide it was a wrong action.
At that point, it seems useful, at least to me, to understand that we live in an imperfect world and may have to do violent things, but can be strong pacifists by asserting that there is never a "legitimate" violence.