This could also apply to moderation in addition to comments; we could make it so only people whose moderations we respect actually count. Others' voting is disregarded.