Even if you remove yourself from the scoreboard, they can still see your karma by just going to your profile, I don't think the name calling would stop.
You know how YC always preaches that you should take your users feed back serious and all that ;)