I agree with you about the voting swings. I think that, more than the extreme numbers, may be why voting is starting to seem such a mob thing lately.

It's a very interesting idea to have users see only the votes of other users they choose. One problem though is that it could easily lead to leaking who voted for what, which would make a lot of users (including me) uncomfortable. E.g. your pool is your 7 coworkers, and user x. One day you know all your coworkers are on a plane, and that you're therefore seeing user x's votes. Another problem is that it could be a lot more expensive to generate pages.