Probably because they are Reddit users who think downvoting is normal behavior. Keep doing what you are doing (upvote the good stuff, ignore the majority of comments and [perhaps] downvote rude and spam comments) and maybe they will go away again...