I very much agree with this. I quite often check the comments before the story on HN. On reddit (programming), the comments are rarely as good as on HN. Also, the turnover of stories on Reddit is much slower.
The very idea that people collect here to talk about a topic at hand means that it's keeps the conversation a bit higher than elsewhere. I don't think I've ever seen people here push their number of upvotes as something that mattered. There is a lot of buy-in from users to ignore joke comments, even if they're on-topic jokes. How the community became like this I can't say since I have only been here about half a year/year I think, but it's a refreshing change of pace when you just want to discuss something.
It's kinda crazy what people here are "sensitive" about! For example, today I have managed to offend Hacker News by questioning the universality of maths. It's a subject on which I am rather qualified to have an opinion, but since it does not match others' I got downvoted. That's just a recipe for homogeneity.
I think downvotes should be removed entirely. Either flag a post for being off-topic, inappropriate or adding nothing of substance to the discussion, or reply with a counter argument to express disagreement to well-written posts that deserve it. The ability to silence opinions different to your own with minimal effort can only hurt the quality of discussion here, in my opinion.