At the very least, if someone down-votes they should reply as to why they disagree. There a great sense of loss when you say something factual and relevant and get down-voted without explanation.
When I've brought these issues up in the past, the conversation usually resolves with "don't worry about karma." And, I don't, but it does change the community, and about that I worry.
Basically, the community will turn into whatever is rewarded. If you reward hive-mindedness, that's what you'll get. If you reward interesting discourse, that's what you'll get.