My general hypothesis is that you can improve comment quality by giving users more metrics than just an upvote or downvote.
For example, consider the following:
Some people are saying there's a lot more negativity around here. I think that's true. Also, I've found that if someone is negative towards me in a way that seems personal, I get defensive and have to work hard to not respond in kind (and sometimes I respond in a mean way anyways). Sometimes these kinds of arguments get a lot of upvotes, because lots of people take one side or the other, and new people get drawn in.
One way to deal with this particular problem would be to treat comments that are "mean" differently from comments that just aren't interesting or insightful or correct.
For example, you could just cut off, or gray out, any comment thread that becomes "mean," thus diverting attention from it; people already involved would be able to continue if they want, knowing nobody else will probably look at it, and new people won't get drawn in. How you detect when a thread becomes "mean" is not easy, but maybe you could have a "mean" button/flag. A person clicks the "mean" flag to vote the thread as mean, but their vote's power is weighted against their karma, and then there is a threshhold that, if reached, actually designates the thread as "mean" and greys it out or whatnot.
Now, the point of this comment is not actually the specific "meanness" example, but the hypothesis I laid out at the beginning of the comment.
I mean, hell, you could try a bunch of different metrics. Just create a little "metrics bar" that allows users to vote on various things (like "meanness," etc.) and rotate a few different options through over time just to see which ones help or give interesting data. Keep the normal voting much more prominent, though, and make this metrics bar less prominent, so that it can be ignored and doesn't impact the UX much for those not interested.