Except that groups have a proven track-record of assembling masses to brigade things they loudly hate. And since most people don't downvote, a determined group of downvoters can send a hyper-loud signal.

Personally, I think this is all flowing from a fundamental problem: there is no concept of "credibility" with upvotes and downvotes. As conservatives are fond of saying "facts don't care about your feelings", but all our social media use feelings to determine which "facts" get the loudest boost.

Any system that uses a raw number of votes (be they up or down) as a weighting score is basically encouraging botnets, brigades, etc. Like, every climate activist has a swarm of climate-change deniers following them around online and I consider their opinions worse than useless.