Absolutely, if you care about those numbers. I think my average karma is pretty low, but that's not why I'm here.
I assume the goal is to encourage high-quality discussions (including extended discussions/back-and-forth), in which case average comment score will work against that.