Comment by PaulHoule | Hacker News Reader

Comment by PaulHoule | Hacker News Reader

Parent Full thread

(1) sentiment analysis is the one area where bag of words really goes to die; there is a limit to how good the results you get will be and it won't be good.

(2) the right way to do this is to train a probability estimator on your scores, that is, put +/- labels on some of your documents, then apply logistic regression.

http://en.wikipedia.org/wiki/Logistic_regression

A lot of machine learning people think this is harder than it is and worry more about regularization, overfitting and such, but in the case of turning a score into a probability estimator you are (a) fitting a small number of variables and (b) if you have a lot of data and make a histogram you will ALWAYS get a logistic curve for any reasonable score, I think it has something to do with the central limit theorem.

This seems to be one of the best kept secrets in machine learning. I used to be the bagman who supplied data to people at the Cornell CS department and we ran into a problem where there was an inbalance in the positive and negative set and in that case the 0 threshold for the SVM is not in the right place because it gets the wrong idea about the prior distribution and T Joachims told us to do the logistic regression trick.

Also if you read the papers about IBM Watson they tried just about everything to fit probability estimators and wound up concluding that logistic regression "just works" almost all the time.

Expanding on the point that bag of words doesn't work that great for sentiment... A good example of why are phrases like "not bad", "not great", "not a good idea". Just scoring based on unigrams really doesn't capture the context well, and people use negators a lot. You could maybe try filtering these out or detecting them with some clever rules.

I did a bit of work on this using the JDPA Sentiment Corpus for my thesis about 5 years ago. Its hand annotated for things like negators and inversions of sentiment. There's a bunch of code and examples here: https://verbs.colorado.edu/jdpacorpus/

Warning: code/corpus is academic licensed, but even reading the papers may give you some ideas.

It is worse than than because of:

(i) negators aren't always next to the word they negate, to get good accuracy you need more of a parse (ii) sentiment is highly dependent on the domain. For instance, if one was looking at people's opinions on stocks, "Buy" and "Sell" are considered sentiment but these are emotionally neutral words in general. (iii) Also there is sarcasm, which sometimes people can't figure out right.

This sounds like a great idea!

I am familiar with logistic regression (studying to be an actuary) but the problem is that my documents are unlabeled: I literally have 2000+ unlabeled documents and a list of positive and negative words.

I'm willing to label 10% (~200) documents but how should I 'score' them? On a scale of [-1,1]? Just {-1,0,1} for negative, neutral, positive? How do I create a training set?

Also can you point me in the direction of some of the implementation details e.g. how do you translate text into logistic regression model?

I would also like to implement POS tagging and 2-grams (e.g. "not bad" != "bad") - any advice on incorporating this into the system?

Thank you for your input!

We had to do something similar in our real world example where we had the labels but were unsure if the labels were truly accurate or not.

We used a technique similar to LSA.

Our first step was to build a bag of words and construct a scaled TF off of that. Then we verified the label for about 10% of the data and used that as our training set. Using cosine similarity (which we calculated using matrix multiplication of tfs) we found top n labeled documents that were similar to document in question to decide the label of remaining 90% documents.

Once we had this dataset we ran it against logistic regression as well by training on same 10% and use remaining to find the label. Interestingly document similarity was only slightly better than logistic regression. Logistic was 10 times faster.

I think this approach worked for us because we had somewhat of a mutually exclusive set of words for one or other label. This may not work in sentiment analysis where same word can have different meanings depending on surrounding words. N-Grams and then TF on it might help in that case.

If you are not using bag of words, then what features are you fitting your logistic regression with?

Reply on news.ycombinator.com