Fair Use is specifically an affirmative defense. Not a defense like self-defense at all. It's not a matter of 'yes I did a bad, but I had a good reason', but 'it is my right to do this, I did nothing wrong'. The other references to Lenz v Universal in this thread established this and it's been upheld in many courts.