I like the idea of not tying the karma score to the user, but that does help identify good users from bad. Maybe cap the karma the user can get from a single comment to something low - say 3, but let the comment itself show the real number of votes.