The voter loses karma. One downvote is one karma. One upvote is one karma. (StackOverflow is 1 upvote for 10 karma for answers, 2 karma for questions; and 1 downvote for 1 karma everywhere, I think.)
The only person to lose karma is the one that gets downvoted.