A time-delay on down-votes perhaps? More of a psychological effect than a technical solution.

Not showing any points above 5 (or 10 or whatever), while still having an internal counter might also help.

A trust system maybe. "I trust this user. So anything he put down as interesting I want marked as interesting."