I was always kind of surprised that there's a relatively high karma threshold for downvoting, but seemingly not for upvoting. Seems like it would invite voting rings. I guess there's already other software to detect that though.
And yes, there a lot of filters in place for upvotes, many of which are intentionally kept secret.
Personally, I don’t use downvotes.