I concur with your proposal and suggest fleshing it out by having 4 directional modifiers:

UP for agree (Propably the most common action)

LEFT for good contribution / insightful / well-constructed argument

DOWN for disagree

RIGHT for bad contribution / troll

This would enable rating of votes in 2 dimensions and may possibly map to a page layout (ie agree+good at top left, then the not so good but still agreed being placed further to the right)

Could alternatively have the comments coloured based on good/bad value (probably some sort of logarithmic scale) and sorted vertically by agree/disagree value, which would still allow conversation threads...