The problem is that this very quickly starts to feel like any other forum where things generally degenerate to the point where only seven people are posting because anyone else becomes afraid of the wolves attacking.
My opinon: Down-voting ought to cost you (the down-voter) a significant chunk of points, say, 10%. And, you should be obligated to post a reason. And, if the reason is bullshit and it gets downvoted, you loose another 10% of your points.
In other words, it better be valid.
The rules could be a little more elaborate than that, but this might be a good start.