Oh, don't get me wrong, I don't believe stuff should be down-voted just because it doesn't fit into the prevailing opinion, I'm just saying that I don't take it personally when it happens.
This leads to cases where coherent and interesting comments are sometimes -4, which is exactly the rating that should be reserved for porn, spam and really gross trolling. At the same time, some users achieve consistently high ratings for comments that seem to be lacking in content, leading me as a new user to suspect that the system is either rigged in some way or at least very susceptible to ridiculous pile-on effects.
The good news is though that this doesn't happen often enough to be a serious problem (yet?).