If you're being disinclined from posting one-word answers and empty "me too" responses: cool. No one would miss those. But if you're stopping yourself from posting interesting or helpful responses to old or unpopular threads? That's when you should stop caring about "karma" the poorly-quantified score and worry more about "karma" the concept. Being directly helpful to a dozen individuals in otherwise ignored threads would usually give a lower karma average than simply posting a clickable version of a link, but you'd get a lot more real respect from it.
Also keep in mind that anyone that sees your average karma will see your total right next to it, and if you game either one too much at the expense of the other it could give a worse overall impression than posting normally. I'd rather see a 1000-total 3-average poster than a 100-total 8-average or a 5000-total 1.1-average.