Personally, I think it starts first with community standards and moderation. Hacker News, Stack Overflow and such do a pretty good job already. You get a share of jerks here and there but by and large the community self-filters the tone[2] as much as the quality of the comments.
When the inevitable trolls appears, one of the best approaches I've seen is to silence them without their knowing it. I wonder how often this practice is figured out and remediated by creating a new account. It'd be possible to extend this further by categorizing troll posts and selectively providing them to each other (as well as providing all replies) so that they think "it's working" and don't bother switching accounts, all the while hiding all of the trash from everyone else. Perhaps that's a lot of wasted energy but I'd be interested in hearing if something like that has been tried anywhere.
[1] That word always seems to describe the behavior best, to me. I'll use "jerk" from now on. :)
[2] I've always believed that you can say "someone/something is wrong on the internet!" without having to be a jerk about it and have seen many examples of such on Hacker News and the StackExchange family of sites. Those who cannot lack creativity and are, unfortunately, less likely to have their (possibly) valid arguments heard.