I think the mods are a larger contributor to the "acceptable discourse" in
most subreddits than drive-by downvoters. Most people with unacceptable opinions (as per the mods) find other subs to congregate in when they've been banned or know their input will result in them being banned.
There are exceptions though: I participate in some capacity in most of the feminist subreddits, and the divisions between different branches are pretty charged. One example "feminismuncensored" subreddit which I believe was created in reaction to the "feminism" subreddit's slant towards white liberal feminism (which has been getting better over the years), but seems to have attracted trans-exclusionary feminists due to the name, and the fact that trans-exclusionary feminist views are also not tolerated in most of the other larger feminist subreddits (for those who don't really pay attention to this, my use of "trans-exclusionary" rather than "gender-critical" is a signal of my own bias towards trans-inclusive feminism).
The /r/feminismuncensored moderators recently stickied a post clarifying that transphobic comments are not tolerated there, and as you can imagine it was heavily downvoted by people who (I think somewhat reasonably) assumed the name of the subreddit meant trans-exclusionary feminism would be tolerated. Comments supporting the moderators' decision there were heavily downvoted as well. So this is one of the (relatively rare IMO) cases where the subreddit appears to be frequented by people whose views are out of line with the moderators' ideas of acceptable discourse. In most other subreddits the moderator bias and community bias tend to be more closely aligned