Being on-topic, polite and factually correct are all virtues that I would expect in a typical HN poster. We should prefer comments which reflect these virtues!
However, regardless of what we intend, I think upvoting and downvoting equate to 'I like' and 'I no like'. We should embrace this de facto reality.
People, including good people with sound judgments, cannot always explain their choices at the time they make them, nor is this desirable.
When people downvote foolishly, in many cases they will have lost their tempers. In such cases sober rules like 'only downvote comments which do not add to the discussion' may go out of the window.
This can't be fixed by exhorting users not to get angry or not to comment when angry, for people can be goaded into anger in subtle, malicious ways (which cannot themselves be easily captured and prevented by other rules). And people can deny that they are angry, even to themselves.
Anyway, effective rules must remain stable under such conditions, which is no mean task.
One way is to engineer new rules into the system itself.
For instance, improvement might be brought about if downvoting, in addition to costing the downvotee one point, cost the downvoter one karma point. People would then refrain from frivolous downvoting, but I believe would still downvote inappropriate, thread-damaging comments. This is partly because good people tend to have more karma and well as better judgment. It also reflects human psychology. Getting angry always costs the angry person.