I think the two obvious metrics are volume of comment votes and average comment score. I don't know which way you would want those two to go though.
If the standard deviation in comment scores decreases, that indicates more people are voting in disagreement with the majority, which is probably the intended purpose.
I believe pg has in the past just used "this seems better/worse" as a metric, but it's hard to say how noticeable any change from this will be.
I'd say the most obvious metric would be whether it evens the distribution of comment scores.