Down-vote away.
It'd be easy to implement a set of rules that could make it fair. For example, limit the rate of change on any given post so that it can't be attacked by a barrage of down votes by fan boys. This would open up a post to a wider non-ideological audience and allow it to float or sink on its merits rather than whether or not the author pandered to the fan-boy culture's point of view on all fronts.
Further to that, nobody should be allowed to down-vote silently. That's chicken-shit. If you down-vote you need to state why you are doing so and subject your view to the same peer review, if you will.
Another interesting idea is that down-voting costs you a significant amount of karma points. And, if you go below a certain number you are ejected from HN. Now people are likely to think twice before down-voting on ideological basis because they'll stand to loose something.
All said, HN is still tolerable but it really is frustrating to be down-voted when purely on ideological basis rather than on based on the merit, accuracy or veracity of a post.
Anyhow, the PayPal, Google, eBay, etc. problem is not likely to be solved by yelling and screaming on blogs and HN. I firmly believe that massive legal action, and, more than likely, in the US, Congressional action, is the only light at the end of the tunnel. They are too big and just can't be hurt or bothered with any other approach.