Well, no, right now average score is meaningless. But if having a higher average score means that you have a stronger voice, as OP suggested, then people will start to care about those numbers.
I assume the goal is to encourage high-quality discussions (including extended discussions/back-and-forth), in which case average comment score will work against that.