Voting should have nothing to do with whether you agree with a comment or not. It should be indicative of the value of the comment to the discussion of the topic at hand.
Update: I will downvote anyone with such stupidly patronizing and passive-aggressive attitude like "if you downvote me (or anyone else) then you are stupid or not worthy". Anyone is free to downvote my comments for any reason they like: be it disagreement, not liking the tone of it, or me being factually wrong. I'd appreciate the correcting comment in the latter case though, but by no means the lack of such I would consider an insult or "slap in the face". I don't think there is any problem with downvoting on HN, but I guess there are problems with some egos.
[Edit: You're right that I phrased my parent comment badly]
The way a comment is phrased is part of this. Coming off like a rude jerk isn't going to help your cause any. I don't downvote things I disagree with, but I will vote down people who aren't helping create a worthwhile discussion.
Should I respond to these? No, I don't think so. What am I going to say? "You're being a jerk, down vote!" doesn't add anything to the conversation either and would deserve a downvote as well.
You can disagree with this approach, but if you downvote someone and then respond, your saying one of two things:
Your comment instigated further discussion on my part, but I don't want other people to read it, so I will downvote simply because I disagree.
Or...
Your comment isn't helping to further the discussion of this topic, but I'm going to respond anyways by telling and as a result create a comment that also doesn't advance the discussion.