In each comment, the karma is bounded by -4. Also, comments can not be downvoted after 12 or 24 hours. (I'm never sure.) So the damage of downvote brigades is bounded.

Also, the system probably can detect it and notify the mods and they fix the karma. (There are a lot of hidden unofficial features. They are part of the secret sauce. Some may be fictional.)

Also, if you see something like this, you can send an email to the mods to they can take a look.