There are just over 600 people in my ward of my township. 67 people voted in 2011 for the Council seat that was up. It's disgusting.
I would posit that the cause is somewhat different, though: the partisan zealots are the ones that vote in every single primary and general election. For those of you who vote for POTUS and that's all, there are actually two elections every year. The country as a whole I think is more moderate than 99% of our elected officials, but the country as a whole isn't voting. It's largely straight-ticket people voting, especially in off years. This results in Members of Congress becoming more extreme because those are the people voting for them.