>I don't yet have sufficient karma to downvote, but personally, I'd like to see all downvotes accompanied with a note or counter-point.
I downvoted you because I disagree with you. Is that the kind of note, you'd like to see clutter up comment threads?
Lots of little notes about why various readers downvoted various comments would be a net loss for the reader. Also, I don't see a reason why downvotes should require any more justification than upvotes. If people hardly ever downvote, then the way to get a lot of karma is to write a lot of comments. If there's anything the internet provides more than enough of it's volume. I'd gladly sacrifice half the commenting volume in exchange for higher comment quality, but an upvote-only system gets the opposite.
Worse still, if downvotes are rare, then many of the top comments will be those that are particularly controversial or divisive since readers who agree will upvote, but many who disagree just don't feel like taking the time to write a note or counterpoint. In the long run, a mostly upvote system will lead to more inside jokes, more inane internet memes and more loud but poorly reasoned assertions.