I don't think users should be penalized for siding with one aspect of an discussion just because other users disagree and proceed to downvote based on their own personal opinion. It's not even the best way of determining majority (which I believe is, partly, the purpose of the system), since not every user makes use of the feature.
Some sort of hybrid might be the most comprehensive solution. Probably something along the lines of upvotes raise karma (compensating good input), downvotes don't affect karma (their purpose becomes, then, to provide insight on the general opinion of a subject), and flags affect karma negatively for rude or uncivilized comments.
Regardless of the original purpose, I think that fomenting conversation should be one of the most prominent ideas behind the whole karma deal. The comments and the conversations are what make HN different (and, of course, demographics, which consequently means that the comments of this given demographic, self-targeted, is what makes HN great), and I know many concur with that statement.
Perhaps this is no longer true?
Regardless, the fact the downvoting is just as easy as upvoting (and has no undo) strikes me as odd.
I suggest downvoting cost the voter something, not just the votee. 1:1 seems simple enough. It would also give greater pause to newer users than veterans. However, an undo (perhaps timed like comment editing) would be more desirable in this case.
I also routinely upvote comments I disagree with but find well-reasoned nonetheless. Especially if they are unusually difficult for me to refute.