Do we know it's directly karma-driven? I'd hope it's more statistical, e.g., downvote to other activity ratio. I'd think that people who downvote very infrequently probably mean it.

On that topic, I'd really love to see downvotes with an accompanying comment weighted somehow.