The quality of comments has been declining very gradually for years. There's a lot we already do about it, and probably other techniques that are yet undiscovered. This is somewhat uncharted territory. I'm medium hopeful.
For example: http://news.ycombinator.com/item?id=3497331
More passion, less thought, less tolerance for conflicting views, worse threads.
Paul Graham seems cautiously optimistic about comment quality. I think he's being too generous. I suspect that advocacy energy is cumulative; it doesn't go away when the topic fades, but sticks around and waits to bind to subsequent advocacy topics, causing the site to steadily crud up with them.
Here we find people of diverse tastes. I am an engineer and, given the possibility, I'd be designing processor architectures and instruction sets. I also like programming - and do it for a living, but I have lost some of my taste for algorithms and data structures and prefer to generate competitive differentials elsewhere on the stack. But I am also very worried by the political background, both here in South America, where I live, in Europe, where my ancestors came from, and with the US, which is a very fine country where a lot of dear friends live and that we shouldn't allow to be ruined by a bunch of crazy people. When I was a teenager, I read 1984 and the idea something like that could happen scared me a lot. It still does. Bad politics is contagious.
I think all is fine, as long as we remain civilized and don't forget to disagree in constructive ways. Community and relationships are important too - I have only one nickname, after all, one which is easily identifiable. All my friends know who I am, where I came from and what my opinions tend to be. I like to think that, in the many times we disagreed about things, we never ourselves a bad example.
Also, as HN gets more readers, it becomes more diverse. I've seen doctors, medical students, lawyers...etc post. It's no surprise somebody like that is going to skip a post about using abstract mathematics to model programming or about optimizing lunch at your startup; they are probably gravitating towards the political articles. On the other hand, just being interested in start-ups and CS does not preclude looking at politics.
So really, I think this is a function of diversity: in a purely democratic system--everybody has an equal vote--less specialized things really have an advantage. This is why I don't bother reading the newspaper or front page of reddit, but still enjoy /r/haskell.
I'm sure there are also plenty of confounding variables; I just think it's more broad appeal than passion. (Seriously, have you read the responses to people's blindly disparaging functional programming? At least as passionate as most political posts ;).)
Think about it. It wasn't like this when Hacker News was smaller, because when HN was smaller there were less people, less comments, and everyone had a voice that could be heard. Now that the place is bigger your "voice" is easily lost, your comment ends up at the bottom, you feel like your opinion won't matter. This stresses you out and pisses you off so you go and piggyback off of someone else's comment near the top and just vent.
We see this in political ralleys, demonstrations, protests. When humans feel they are NOT being listened to, they get loud and angry.
I'd strongly suggesting policing OT posts more proactively. Like PG said in his essay about starting Hacker News, this is a social problem, not a technical one. We need to refrain from doing things that make jerks feel like this is the place for them.
My question was asking if there may be a correlation between the quality of the content and the actual name of Hacker News being misrepresentative due to the ambiguity in the definition of "hacker."
Which I asked because my thoughts seemed to wander a bit too far from what you were asking.
It seems that hacker news may be getting too popular for it's own good. At which point, you start getting submissions from people who are more interested in promoting themselves or some agenda and finding ways to game the ranking system to get content on the front page.
Now, I think it has become too large and feels anonymous. Posting something offensive has no cost because it doesn't feel like a "know your neighbor"-type community anymore. The costs imposed by karma seem to be less significant. I even think that if you post a rude or ideologically motivated comment, there are other rude people who will agree with you, attenuating the negative signal.
The HN is turning to reddit mime is genuine. I think it's just a consequence of having a very large community. Although a lot of HNers may disagree, I think the herd must be culled. I'd be happier of participation was limited. If I was the benevolent dictator, I'd collect a list of people whose contributions are most valuable and remove everyone else. Then, give those high-signal / low-noise contributors the ability to invite other people. Continue to have all content publicly readable, but ration the writes.
(Disclaimer: Although I have many doubts, I do believe it is possible to have an HN-scale community that remains civil; however, without the data, I'm not very comfortable just enumerating implementation ideas.)
Different subforums have their respective groups, which you learn once you spend some time inside the subforum. Despite being a huge ecosystem, they have managed to keep many of the subforums feeling small.
On HN...I rarely notice who is posting. A few influential posters stand out but for the most part I do not remember names well. Everyone blends together.
Perhaps something should be done to alter the direction that the content is moving, too. I personally agree with tptacek that gossipy advocacy/techpolitic issues are now far too important to the site. I enjoyed them a little, too, but I feel like the community is now addicted to them. Honestly I don't want to see anymore gossip pieces from torrentfreak...
* For example, (1) give people limits on how often they can post while their average karma is low, (2) allow people to flag for more reasons (hivemind posts, aggressive posts, bigotry, rudeness), (3) give people a couple of weeks/months to time-out from the site if they get flagged and there turns out to be a problem with their behaviour. I think there is a danger on a site like this of people over-complicating the problem when there are simple things you can try. :)
It seems like an extremely difficult problem, and relevant to the problems many of us are trying to solve. I can come up with plenty of simple metrics, but they all fall apart pretty fast, especially if the community quality starts to tank hard. Think of the sheer number of upvotes that most inane meme references get on reddit.
Situations like an inflammatory comment sparking an insightful debate are one of the things that initially seems like a poor quality comment (and it quite likely is), but overall might be a positive thing for the community.
Quality is discourse, not peanut gallery, vacuum sucking commentary that builds on puns, memes, insults and karma whoring.
Inflammatory comments often devolve into nothing. No insights, no evocative call to action. However, constructive assertions backed by reason are not inflammatory and equally capable of evocative statements that can be challenged.
What I'd love to dig in around, though, is if there's a way to quantify the difference between an inflammatory comment vs. a constructive comment.
Just thinking out loud: Source links/references perhaps? If it's a primarily English-speaking community, look for specific keywords? As a negative metric, look for an extended back-and-forth between two people?
This is one of the things I'm struggling with - it's easy to come up with "dumb" metrics & ideas, but the false positives/negatives are a real problem.
I think that HN is set up to only be as 'good' as the users, and I know that this is true of any community, but there are still issues that often spur from differing opinions no matter how awesome the community is. I believe that the best way to combat something that you don't approve of is to just ignore it. I think that the comments that people consider unfit for HN are few and when ignored will make the poster consider changing his posting technique to gain more popularity and give weight to his words, because after all we all want to be heard. That being said, is the present that much different from the past?
Quora, which spoke very openly about the problem of mantaining a high standard of quality in spite of growth, is a prime example since the structured Q&A format makes inadequacies even more evident; as of today, it seems hard not to think that their efforts could simply fail. (If you are interested in this specific case, take a look at http://www.readwriteweb.com/archives/how_can_quora_balance_q... or, for a longer read, http://www.wired.com/magazine/2011/04/ff_quora/all/1)
Yes I realize there is always the inevitable discussion of "HN is declining" and like most things, perception is reality.