I handle the downvoted comments by 1) upvoting (but of course) but then also 2) adding a comment to the effect of "Hey, I upvoted you because ...".
We've all seen the downvoted comment start to "fade back in" by doing this.
Heck, I frequently even upvote greyed out comments I disagree with because I kind of root for the underdog anyway. Or, kind of as you say, think it is unfair (at least when the downvoted comment was sincere and not troll-bait or whatever).
Personally I try to avoid downvoting a comment if I am unwilling to leave a comment as to why. This open the doors to my getting downvotes as well if I am off base.
Regardless, watching my own comments get downvoted has been a good lesson for me. Sometimes I rethink my position ("Am I missing something here?") or, if nothing else, I rethink the tone I used ("Guess I need to make a better case next time — not come across so antagonistic.").
(Corny examples, but you get the idea.)