On the inverse side, I think I'll also track your voting rates, and if you tend to rate too many people as trolls or spam then your voting will also be throttled.
But, it's also just a plan. The goal is to create something that lets communities moderate themselves without too much tyranny of the majority of minority. That's why the code is open, so that people with ideas about how to do that can help make it happen.