I'm not sure what scenario you have in mind, in which the thing they have been annoyed by would be useful as a defence.
Normally this discussion arises before the bad thing has been done, or if you weren't involved and found out about it. If you in fact committed it on their behalf without telling them first, then you are actually the bad guy.