What about this: to to downvote, you need to also specify a keyword as to why you are downvoting. It's a text field (not a pull down). Whatever you enter, will not be visible to anyone except to the person who wrote the comment you are downvoting.

Adds just enough friction to reduce downvotes a bit, provides information to the person being downvoted (no more "why all the hate?" comments), and doesn't pollute the experience for everyone else, since those are not visible by most users.