I'll consider a comment for a down-vote if it negatively affects the conversation. Usually this means it's off-topic for the conversation that it replies to (including insults and just plain meanness).

However, when I determine whether to down-vote, I first figure out what I think the score should be, and only down-vote if it's above that value. I've even up-voted comments that I thought were bad because I felt the author was being punished too harshly for saying something they obviously didn't realize would be bad.