If it requires a comment to downvote, then it should be the same to upvote.
I've been genuinely curious why some of my comments have been hammered, and an indicator would be helpful, but keeping the feedback loop simple is likely to increase the usage.