Do Social News Sites Deteriorate? (Analysis of 1.8M HN Comments)

Actually it's because the site has become so large that I have to be more diplomatic. When it was small I could say harsh-sounding things without worrying about being misinterpreted. Now I have to be more of a politician.

So there seems to be a lot of skepticism about whether our algorithm can actually measure emotional impact accurately. For the long answer, I'll refer you to the about page[1] for EffectCheck. For the short answer:

My co-founders are an AI PhD and a Clinical Psychology PhD. They spent three years curating a huge dictionary of words using a methodology similar to the Harvard Psychosocial Dictionary [2], but with the twist that they were focused on lexical impact of words. The dictionary is pretty accurate at measuring both impact and sentiment [3]. For example, we can predict Amazon reviews as being positive or negative, using the stars to validate if we are correct-- blog post on that coming soon.

Regarding context of the word usage: Both behavioral studies and fMRI scans have confirmed that context is not as important as one might believe. Our brains process multiple meanings of words in parallel, and the emotions associated with those words linger in our subconscious even after we know the correct context. Similarly, in cases of reviews and comments, people who use hostility-evoking words are often hostile themselves (angry people tend to make others angry) and the same is true of the other five fundamental emotions we measure.

Happy to answer more questions. Also happy to analyze any data that you would like to see in order to verify accuracy of the algorithm-- just give me a link or the text. :)

[1] http://effectcheck.com/about

[2] http://www.wjh.harvard.edu/~inquirer/homecat.htm

[3] Note that sentiment is correlated to impact only if the writer or speaker is writing without detailed attention to word choice. For example, political speech writers comb over every word to make sure they have the desired impact-- thus, the final text likely has little correlation to the sentiment of the orator.

It would certainly help if the algorithm used to compute these metrics were shared and dissected. I tend to believe that sentiment analysis is much more art than science, given how tricky and profound context can be.

To conclude, I'm not going to end this comment on a negative note. I'd much rather reserve my vitriol and caustic criticism for another thread and day and abstain from calling this an attempt at gaming HN to promote a startup.

I do this not out of a lack of indignation towards what I've just read, but because I'd like to end this seemingly hateful comment on a note that isn't bitter or negative but is instead quite the opposite (without using a single word that would help the OP's algorithm figure this out).

And with that, I throw down the gauntlet. Analyze this!

In other words, I wonder if these graphs are just tracking general sentiment, and would look the same for any site.

For example, some emotive words that appear to have increased since the beginning:

* sad: http://hn-trends.heroku.com/trends?q=sad

* fuck: http://hn-trends.heroku.com/trends?q=fuck

* shit: http://hn-trends.heroku.com/trends?q=shit

Some words that appear to have decreased:

* great: http://hn-trends.heroku.com/trends?q=great

* cool: http://hn-trends.heroku.com/trends?q=cool

* lol: http://hn-trends.heroku.com/trends?q=lol

* reddit: http://hn-trends.heroku.com/trends?q=reddit

* digg: http://hn-trends.heroku.com/trends?q=digg

I haven't looked at any of this closely, so make of it what you will.

(I was meaning to wait until I had time to dig deeper and add extra features like time series smoothers and trend/slope metrics before "releasing" this, but figured this might be useful now given the post. I still plan on adding those later.)

1.He's proposed a full service taxonomy with by proposing a name for the phenomenon SND. Which is much a much better choice than JTS - Jumping the Shark or otherwise.

2. He's asking how we can evaluate that phenomenon. Proposing one solution.

So the question becomes, how else can we evaluate the phenomenon and what can we do to reduce SND?

Well having spent time elsewhere, here are a number of clear indicators of SND:

1. Shorter, less thoughtful responses, often veering into humor or the absurd. With chuckles getting the most upvotes

2. Less fact checking and less source linking in both posts and comments

3. More image / pic posting

4. Linkjacking, with materials not linked to the original materials.

5. More community centered posts aka AMA etc.

6. Fewer news links.

So perhaps simply evaluating the length of comments of that same 1.8M HN data could support PG allegations.

The next question is what can be done to prevent SND?

I think that would be clear: Don't support the characteristics that lead to the decline.

Hypothesis: The net mood of HNer's is deteriorating because it the ratio of financially successful members is lowered as a more people join. In addition as HN grows the ratio of preexisting and external relationships among members gets smaller, e.g. the number of HN'ers affiliated with YC companies has not kept pace with the overall growth in HN membership.

http://news.ycombinator.com/item?id=2059012

(now back to peaceful slumber)

BTW, I think what you're doing is brilliant, and I can think of about a dozen practical applications for your technology.. Ignore the naysayers.

I would challenge the assumption that the symptoms of SND include a decrease in the positive emotion measured in the average HN comment.

Is it not possible that there is a growing segment of the HN user base which posts and enjoys the kind of comment that induces negative sentiment in people like pg? Would that not raise the sentiment of the average HN comment, while at the same time lowering the sentiment of the average "good old days" user?

>There is a perceived phenomenon among online social news communities (e.g. Digg, reddit) that as the popularity of the community increases, the average member becomes baser and the overall quality of discourse decreases.

I think that's not an accurate portrayal. Having been through the "decline" of Reddit (been reading it even before it had comments!), I observed that the real decline was in the quality(as perceived by me) of stories that made the frontpage(s) and in the quality(as perceived by me) of comments that were upvoted.

In 2006, I would literally read almost every story on the first few pages and the content was very interesting, there were long essays and articles that were on the front page. Now? After the influx of the general public and the juveniles(not saying this in a demeaning manner), quality has dropped in every way. Pun and argumentative threads were the ones that were on top. I do like humor but not just endless pages of snark and humor. Pictures or story-in-headline posts requiring an attention span of 3 seconds tend to get the most votes,

>If a community were suffering from SND, there are a few symptoms we might observe:

>Anxiety, hostility, and depression would rise.

>Confidence, compassion, and happiness would fall.

>The ratios of anxiety/hostility, hostility/compassion, depression/happiness would rise.

>Since each of the negative emotions has an opposite emotion, the last point enables us to measure the general negative/positive trend in each of the three main categories.

How does this analysis address the above points? All the supposed-to-be-funny short comments probably would count as a positive under this analysis. Reddit's community now is pretty happy, confident and compassionate. The analysis is pretty lacking when applied to a supposed-to-be intellectually stimulating social site.

Anyway I see that Slashdot(although not as popular now) and HN seem to have bucked the trend. Although there is some decline on HN, it's not as as steep or as bad as Reddit so far.

Edit: Here's an idea, do your analysis of all the comments of the first x posts of Reddit vs. the same on HN. Could be interesting since they both have almost the exact same format.