Although pg hasn't implemented a change to this effect, I believe he would agree that -- "people feeling they have to go out of their way to vote them down" -- implies a downvote should be more work than an upvote.
I think a downvote should require a response to the parent comment that:
1. contains at least 15 English words, excluding quoted phrases
2. quotes at least min(25% of entire comment, 5 consecutive words) of the parent comment
3. meets current downvote requirements (karma and downvote bombing protections spring to mind immediately)
For the sake of argument, such a "required downvote comment" could contribute almost nothing -- but it would still reduce downvotes. An example of a really bad downvote comment on this comment might be:
> a change to this effect
This is HN, not reddit. Um, did you read the article? I down voted you.
The 15 English word requirement sort of requires some sort of explanation. Does anyone think this would help?