Hacker News Reader

Comment by Profan | Hacker News Reader

If you haven't yet attempted to build some sort of sentiment analysis by yourself yet, be it rule-based or on statistical analysis, you should, even just a rudimentary rule based one is a lot of fun to implement, and it works surprisingly well [0].

One of the harder parts of making a decent one based on statistical analysis however is the lack of good training data, other than the analyzed twitter dataset [1] and another movie reviews one [2].

[0] http://fjavieralba.com/basic-sentiment-analysis-with-python....

[1] http://help.sentiment140.com/for-students/

[2] http://www.cs.cornell.edu/people/pabo/movie-review-data/

View on HN

Good training data's partly hard to come by because there's often reasonably poor inter-annotator agreement on sentiment datasets -- that is to say, humans disagree a lot in how we interpret a phrase. What reads like sarcasm to you might read like genuine enthusiasm to another.

It's pretty easy to load up a set of data into a crowdsourcing tool and use microtasks to rate it, but my experiences doing so weren't superb (even restricting to native English speakers alone).

A better source of data is starred reviews where you have the star rating and the review itself -- these come free with a sentiment rating, although plenty of caveats around normalization. There are lots of places with review systems like this and some (like Yelp) even make the data available: https://www.yelp.com/academic_dataset

I wasn't aware that yelp provided a dataset, that's very interesting!

Since I had this very problem as I was working on using the output from sentiment analysis to modify sentences so to invert the sentiment polarity (positive to negative, negative to positive), the datasets I found were never general enough (movie reviews, many domain specific terms, hard in the text generation step), or had a lot of noise (twitter dataset).

Though evaluating the system was very hard, due to the reasons you stated, inter-annotator agreement was beyond terrible.

I'll have to look into if other review services expose their data as well, seems appropriate.

Reply on news.ycombinator.com