Downvoting needs to cost a karma point, or perhaps a fraction of one.
Such behaviour would appear to be negative if it cost karma. Sadly social conditioning and moderately low self-esteem mean that this would moderate my application of this [apparently to me] beneficial behaviour.
Social sanctions like that may be necessary, but they also degrade the quality of life overall. Downvoting should have a cost, especially when it's an anonymous act.