I think the best way to handle this is to apply a sort of social equivalent to the PageRank algorithm to users. Thus, when a user votes up/down an article, its points change as a function of that user's karma level. Consequently, users with greater karma have greater direct input into the community.
Eigenvalue centrality is a fascinating algorithm that provides such a use case: http://en.wikipedia.org/wiki/Centrality#Eigenvector_centrali...
Another benefit of such a system is you would no longer need to enforce karma thresholds for downvoting, as this sort of meritocracy arises naturally from the algorithm.