So...
How about hiding the points until after you vote. That way, you're not influenced, but you still get feedback that is necessary to keep you voting.
The question I'd ask you is what matters more, the discussion or the moderation? Shouldn't the attraction to the people commenting here be the quality of the conversation, rather than the ability to vote things up and down? The latter is important, but does it have to be seen?
* I often don't vote
* to see how, exactly, the community feels about it
* to see if I should actually read it
* to get a feel for the story's exposure (more votes, more eyeballs)
This wouldn't be an issue if it weren't for the first point... I don't want to have to vote on each comment to reap the benefits of the other items on the list.
The second item on the list allows me to have the "blink" effect. By taking in the vote #'s of hundreds or thousands of comments, I just get a "feel" for what HN likes and dislikes. To me, that's it's important.
I think something in between would be a color-coded and/or normalized indicator of each comment's popularity. Kind of like how the grayed out text is more intense for more downvoted comments... upvoted comments could have some indicator of how popular they are.
If normalized, it would show how popular they are, but only relative to other comments with the same parent as them.
Just my $.02