If you see your karma going down its supposed to be an indication that you are not contributing in a positive manner not that people disagree with your point of view.
We almost need votes on votes to indicate if the vote is constructive or destructive, although that's quite meta. (in all seriousness though there is probably a statistical analysis that can spot the different types of vote)
Sometimes the destructive behavior is actual and embodied in the comment being downvoted.
Sometimes the destructive behavior is potential and in the person down voting.
I've downvoted bad comments. I've downvoted comments instead of arguing with them. I've downvoted comments because I was feeling ill-will toward the ideas it expressed, and I've even downvoted comments because I didn't hold good-will toward the poster.
In each of these cases, I had more destructive alternatives to the down vote. Was it fair? Well when a person has been around long enough, the recognize that these things even themselves out over time.
BTW, a community in which everyone votes in assent and descent according to some set of rules, isn't really voting.
This would help curb abusive down-voting. Too many people downvote whimsically simply because they disagree.
I know that might sound trite, and it's obviously not the only factor, but I bet if karma went away, you would see a measurable decline in submissions over time.
You can get 100s of karma and then get banned for one post that gets blasted for going against the hive mind. (happened to me 3 times).