I think the problem is that some people are not as thoughtful as you are when it comes to up/downvotes. The terrible "downvote because I disagree" is not as bad as in other forums but it's definitely on the rise. When people use the voting system to express an opinion instead of judging the quality of the comment itself it kills the quality of the discussion very very quickly in my experience.

The people who don't take the voting system seriously are probably those who are the most likely to mindlessly upvote/downvote stories and comments.

I think up and down votes should be rationed. Or at least have some kind of cost (when your ratio is depleted you can upvote by using your own karma?). I remember slashdot had (has?) a similar system. It would force the users to think twice before mindlessly up or down voting anything. Maybe a downvote should be more expensive as an upvote, even.

Flagging stories would still be free for completely off topic contents (and fighting voting rings).

EDIT: or as an alternative: the more you vote the less your vote is meaningful.