I don't think users should be penalized for siding with one aspect of an discussion just because other users disagree and proceed to downvote based on their own personal opinion. It's not even the best way of determining majority (which I believe is, partly, the purpose of the system), since not every user makes use of the feature.
Some sort of hybrid might be the most comprehensive solution. Probably something along the lines of upvotes raise karma (compensating good input), downvotes don't affect karma (their purpose becomes, then, to provide insight on the general opinion of a subject), and flags affect karma negatively for rude or uncivilized comments.