I notice that people care about it and experience consternation about downvotes, I don't yet understand why. In my experience with the site, every comment speaks for itself, and getting downvoted just means a lot of people disagree with you. When it happens in real life it doesn't bother me, I don't see why it would on here.
The thing I'm really looking out for when I browse are the topical mega-comments that appear sometime, usually when someone understands the topic at hand better than the original author. There are only so many of those to go around, so the rest is chatter, it has to be. Everyone wants to talk, only a few people have relavant insight... still true to life :S.