This is one of the flaws of the upvote/downvote system, it can give people a bit of a mob mentality where making a comment or opinion disappear is as easy as a click (and anonymously too).

Maybe downvotes could cost the voter karma, or have a higher bar for who can downvote.