A little example: I’ve used the site for a little over 4 years but don’t usually post or comment. If I do comment, it’s something really specific or to try and help clear a misunderstanding so usually low karma amounts. I regularly participate through flagging and upvotes but only have ~50 karma so I still can’t downvote and will likely never be able to.
Not saying it shouldn’t be this way, but downvotes are heavily biased toward people who regularly post or comment.