Downvotes are supposed to be for "comments that don't add anything to the discussion" - personal attacks, comments that are factually incorrect, comments that don't have any content, comments or commenters who have an axe to grind and reply with the same screed to every story, etc. You can disagree with a comment and still believe that it makes a valuable point, eg. if it raises new information or a different perspective that makes you think, but then after you've thought about it, you still believe the argument was unpersuasive.
I agree with you, though, that it's better not to do so. A downvote is one bit of information (literally). An intelligent comment expressing the reason for disagreement is much better.
Even for factually incorrect comments, evidence of the incorrectness is better than just a downvote.
But for personal attacks, axe-grinders, and trolling, sure, downvote them. Absolutely.
If people replies to me, I typically upvote even if I disagree because I appreciate them taking the time to respond to me. If I disagree, I prefer to comment again.