Or what about opening up down-votes to all users again, and then weighting the value of down-votes (karma level) rather than up-votes.