Okay, here's a trick. The goal is to detect people who can't discriminate bad stuff from good, can't restrain themselves from replying to a troll (or can't identify when they are doing so).
The mechanism is, counter-intuitively, to allow people to rate their own comments. However, this rating is not to be directly used to affect how the comment is displayed, since that would obviously be easy to game. Instead, it would be used to identify people who can't restrain themselves from engaging in a low quality discussion; ie, replying to a troll or similar.
Obviously we can't expect trolls to rate their own post as trollish, but we can expect at least some people to have the self-knowledge required to admit when they are posting while annoyed or similar.
So, the system assumes that a comment is only as good quality as its parent, or at least, not much better. Obviously, one can make a good contribution in reply to a bad one, but it's not necessarily a problem to discourage this - since the reader of such a contribution has to read the bad contribution to get to the good one.
Internally, each user has their own scale, which their replies - and the replies to their comments - relate to the scales of other users, using the above assumption. A user would then be shown only comments to which they are likely to reply; this means trolls see all the trolls, but quality users see only other quality comments.
There are a couple of potential problems. One is that it would bias people against replying to deeply nested comments; and ultimately, towards making submissions instead of replies. One rather drastic way to deal with this would be to remove the submission mechanism, and only have one conversation tree; the frontpage items would then be the roots of current discussions inferred from the structure of the tree.
The second problem is that while this mechanism would help current users filter out trolls, it would do nothing to help new users find the best discussions. There is a conflict between preventing the system being gamed, and allowing new users to find the best discussions; because the former would prefer the ratings to be secret, but the latter would allow information about them to be public. It is not obvious that there is a workable trade-off.