I don't know if it's still true, but it used to be the case that if your reply to a comment your downvote to that comment does not count. It was a pretty clever mechanism in that you could be certain any downvotes didn't come from people replying to you.

I don’t see why HN doesn’t require every down vote to have a comment. Judging from the sheets of gray on most threads, most down voted comments aren’t actually in violation of site policy, off topic, trolling, etc. It’s just a few little people disagreed. Which is fine! But it’s not constructive or interesting to just slap the commenter with your dick.