Adding to this: Replace the down vote icon completely with the flagging icon, as the down vote icon probably encourages aggravated people to use it in the wrong way.
My own personal guideline: upvote = "I would like more comments like this" and downvote = "I would like less comments like this". My reason for wanting less comments of a particular type can be a lot of things -- incivility, factual incorrectness, being off-topic or distracting, being technically correct but misleading, or even being repetitive in relation to prior comments that already made the same point better.
https://news.ycombinator.com/item?id=117171
https://news.ycombinator.com/item?id=658683
Here's pg trying to address incorrect downvoting: https://news.ycombinator.com/item?id=1057338
I think pg is wrong in this next comment, I think people vote to disagree, and I think there's a pile-on effect. But I don't have the data, and he (and dang) did (and do). https://news.ycombinator.com/item?id=2403716
EDIT: Some of those threads talk about the "meanness" of HN. I know I'm sometimes dumb and sometimes mean, so please if you see me doing that downvote me, flag me, send me an email.
It's clear that some people are using downvotes aggressively, and that needs managed. I actually had a period of time when every comment I posted (no matter what time of day or what topic) received exactly one downvote between 5 and 6 pm in my time zone, as if I'd offended someone and they were retaliating. But I've also seen people post comments that are purely inflammatory with no worthwhile content (along the lines of "[group] is evil", where [group] is mainstream -- not ISIS or Nazis, but like, Republicans or Muslims) that didn't get downvoted. So the issue is more than "too few downvotes" or "too many downvotes".
So I try to be part of HN's corrective mechanism. If I see something blatantly inappropriate, I downvote and flag if appropriate. If I see something reasonable that's grayed out, I upvote. If I see something non-gray that I think is of negative value, I downvote, and I hope that if I misjudged it someone else will come along and cancel out my vote.
> Downvoting for disagreement is not always bad, but it sometimes is. There should be some nuance here. Don't be indiscriminate. When a comment is blatantly false, downvoting is probably ok, especially when there's something else wrong with the comment. But high-quality, polite posts don't deserve penalizing just because you don't share their position. When you see one of those in negative territory, please be a good citizen and upvote it back to par. Users doing that is one of the community's self-correcting mechanisms, and it's more important now that we've made some downvotes more powerful.
(Emphasis mine)
For what it's worth, I agree that this should be in the site rules.