Deeply Moving: Deep Learning for Sentiment Analysis | Hacker News Reader

Deeply Moving: Deep Learning for Sentiment Analysis

nlp.stanford.edu

173 pointsninjin45 comments

If you're interested in using deep learning for nlp, I highly suggest putting a pot of coffee on and watching this a couple times: http://www.socher.org/index.php/DeepLearningTutorial/DeepLea...

The really neat part is moving away from bag of words representations. Bag of words was always a bit of a hack. But seriously, watch those videos, it will give you a good idea on how it all works, the op is using similar methods.

Edit: thanks gallamine, I fixed the link

Dave, you're missing the ending "l" on the URL.

Thank you so much for this, I'm just getting going in deep learning. I've been viewing and reading Hinton's stuff, and found it tough to get traction (I am just starting, that is not intended to be a comment on Hinton, but on me).

I know there's deep hate for java around here, but here's another relevant link. (You could probably shoehorn in scala or clojure as well) This is a direct tutorial for doing deep learning.

http://nlp.stanford.edu/courses/NAACL2013/

Take a look at a recent homework assignment they had to do with Named Entity Recognition and Deep Learning: http://nlp.stanford.edu/~socherr/pa4_ner.pdf

It's good to see neural networks being leveraged as they are. Conditional Random Fields have been dominant for sequence based classifiers for a while. We've been needing to push the envelope a bit further.

This is pretty cool. I love how in some sentences, the actual words or phrases are all marked positive, yet it is able to accurately mark the entire sentence gets correctly marked as negative. For example, from the paypal/mailpile article from the front page:

Are the risks larger because we are successful?

"larger because we are" is marked as positive, "successful" is marked as very positive, the rest are marked as neutral. But the entire sentence is marked as negative (which to me it certainly reads negative).

How is being successful negative? If anything it's neutral, being successful is clearly good, having exposure to larger risk is negative, a net neutral.

Being successful is positive, but in the context of the article (paypal not releasing their money) the statement is negative.

Fascinating project, I worked for a data analytics company and we always had a basic test that we'd try to throw at any Sentiment analysis engine.

// Negative > This is shit.

// Positive > This is the shit.

Most engines can't sort that out. I'm definitely going to take a more in-depth look at this.

It doesn't look like this one was able to sort those two out either. It gave me strongly neutral signals for both sentences, but that may be because 'shit' isn't currently tagged.

"This is the shit" is a very specific idiom. Unless the engine has been specifically trained to understand it, it is not surprising that it doesn't know what to do with it.

"This is shit" isn't inherently negative. Consider the question "what is the brown substance in the toilet?" and you can answer a neutral "this is shit".

Presumably if you throw a realistic corpus of written text you will have much more occurrences of the idiom.

Sentiment analysis implies attitudes, which imply context. A robust sentiment analysis system must either be trained on a targeted corpus (in which entities/concepts and attitudes are well aligned) or include some way of establishing a more targeted context after having been trained on a general-purpose corpus.

That is to say, people generally aren't describing the physical material occupying a toilet in a corpus of movie reviews.

I was referring to the previous comment, where the two lines about shit were used as a test for any sentiment analysis engine. Your point about context is what I was trying to illustrate.

I imagine that someone using the word 'shit' to describe the brown stuff in the toilet is probably negatively inclined towards it. Someone discussing it in a more neutral context (like a doctor or scientist) would probably be more likely to say 'feces'

FYI, since it's not exactly obvious: in the live demo (http://nlp.stanford.edu:8080/sentiment/rntnDemo.html), you can double-click on the tree image that it generates to see a zoomed-in version with more information, and correct improperly labeled nodes to help the algorithm learn. Try it with some Rotten Tomatoes review snippets - from the ones I tried, it got almost all of them right.

Also, they have a demo that supplies you with a sentence that you can check and correct in order to teach the model [1]. It was actually easy and nice enough to keep me there for quite some time giving me a few nods/laughs and forcing me to think about the sentiment structure of the sentences. The only request I would have for a feature is a "skip" button, since at times I am not entirely sure in regards to the sentiment of a sentence (you can still reload the page though and it will have the same effect).

[1]: http://nlp.stanford.edu:8080/sentiment/labeling.html

It's interesting how the decades of research in computational linguistics still come down to rating a single sentence + or -. And though they are getting better, the systems are still pretty open to 'obvious' attacks:

> I have seen many good movies, this one is not one of them.

> All movies, except this one, are good.

> I thought this movie was going to be bad, but I was wrong.

which rate positive, positive and negative.

I'm full of hope that the modern neural net and linear algebra type methods will eventually crack this though!

Another hard problem: sarcasm. it's fun to think about how to approach it.

Yeah, so much fun.

Clever. hehe That aside, has anyone saw any recent works on detecting sarcasm? There's not much advancement in this that I've seen.

This deep learning method is an example from going from bag-of-words to document level analysis. To detect sarcasm, you'd have to make another leap from document analysis to personality modeling. Humans can detect sarcasm because they^H^H^H^Hwe can build a mental model of what is expected to be said—by a particular author or about a specific topic. If a statement is sufficiently contradictory to our mental model and we are sufficiently confident of the predicted sentiment, we know it's sarcasm.

Without knowing who said something or in what context it was said, even humans would fail to accurately detect sarcasm. Here are some hasty examples that could vary, depending on the author or situation:

"Michael Bay really outdid himself on that one." "That Kanye West song is so well-written." "It'd fun to build that PHP."

Very good point. I love the way you framed that. I took some very cursory attempts at doing it at the sentence level based on ngram representations, obviously this was limited.

You also bring up a very good point about humans being able to barely detect it. It's hard to infer from text without tone a good portion of the time.

It'd be neat to see what we could do with tone as an input feature (obviously we couldn't get that most of the time) but maybe if enough tech becomes available that voice inputs are common, sarcasm can be a relevant enough problem to solve.

In general, I know it's not really worth it to try to solve (that I could think of immediately anyways) aside from "just because".

That being said: ambiguity is ambiguous. What can you do aside from approximate as best you can?

Structured sentiment analysis has been around for a while - meaning models that take into account the overall document as more than just a bag of words (reference below to a 2007 Google paper). There are a surprising number of challenges for those new to the problem, e.g. sarcasm or idioms. It's kind of fun to work on.

http://acl.ldc.upenn.edu/P/P07/P07-1055.pdf

Ethical considerations have always kept me from investing too much time in areas of computational linguistics that seem to be predominantly used for the purposes of rapidly commercialised and now widespread mass surveillance system development. While it is an interesting area, in fact it represents a change to society that I would personally feel happy not to have accelerated. Not sure if anyone else has an opinion on the ethics, here.

I would like to get an accuraye overall result of the sentence. For example this "I found the result of this study amusing since it was far from accurate." was classified as nothing less than neutral but I was trying to see if it would pick up the sarcasm. Is sentiment analysis not yet there?

New Ruby module on Github to import & parse the treebank corpus: https://news.ycombinator.com/item?id=6395420

Does anyone familiar with NLP know how the sentence trees are parsed? Can something like python's nltk do that? I always assumed that was very difficult to do well.

The sentence trees are parsed with the Stanford NLP Parser, available here: http://nlp.stanford.edu/software/lex-parser.shtml

Cool. Thank you!

Some out-of-the-box parsing can be done with NLTK and OpenNLP. Just be mindful of the fact that the quality of the parsing is fairly dependent on whether the corpora you provide are sufficiently similar to the corpora that the models (part-of-speech, parsing, chunking, etc.) were trained on.

Parsing is one of the most competitive areas of research in NLP and yes, you are correct, doing it well is very difficult. State-of-the-art performance on newswire data is around 95% if you score local structure.

miket has already replied with a well-established lexical parser. The Stanford parser is one of the most robust out there, you could also go for the BLLIP (or Charniak or Charniak and Johnson) re-ranking parser [1] that although a bit old and fiddly (the GitHub version is very solid though) tends to perform well.

However, both the Stanford and BLLIP parsers are constituency parsers (now we are going to go all linguistic here) and there is an alternate paradigm of dependency grammars [2]. I have heard a couple of times that Google is running a variant of the MaltParser [3] in-house and it is a very solid piece of work (the command-line interface requires some patience though, but there is plenty of documentation). People for example in Information Extraction (IE) tends to favour dependency grammars, probably since you get relevant modifiers closer to your point of interest in the tree.

Now, we could also go into richer grammars like Combinatory Categorial Grammar (CCG) [4] and Head-driven Phrase Structure Grammar (HPSG) [5] but I think this reply is long enough by now...

Personally I favour dependency parsing since there is more annotated data for it across plenty of languages. Also, the paradigm can cope with a lot of linguistic wonkiness that is very much present in language other than English (free word order and non-projective structures for example)

[1]: https://github.com/dmcc/bllip-parser

[2]: http://en.wikipedia.org/wiki/Dependency_grammar

[3]: http://maltparser.org/

[4]: http://en.wikipedia.org/wiki/Combinatory_categorial_grammar

[5]: http://en.wikipedia.org/wiki/Head-driven_phrase_structure_gr...

thank you ninjin

It doesn't seem to understand valence shifters. It seems to have been trained on not good, but not bad is counted as negative too.

Sentiment analysis has never been this good. Sentiment analysis has never been good.

Wasn't there a company that did a similar sentiment analysis on Twitter and wanted to trade the stock exchanges on that information? I wonder what happened to that.

You mean, stocktwits?

You might be thinking of Derwent Capital - http://www.bloomberg.com/news/2013-02-04/dcm-capital-puts-it...

"In constrast, our new deep learning model actually builds up a representation of whole sentences based on the sentence structure. It computes the sentiment based on how words compose the meaning of longer phrases. This way, the model is not as easily fooled as previous models."

Wait, that is all that's necessary to invoke the term "deep learning"?? Wow. Wikipedia seems to agree. I had thought there was something more to this buzzword than "slightly less shallow heuristic guesswork than the AI you already know and love".

To possibly clarify, the way sentences are structured is itself learned, rather than relying on hard-coded rules.

Maybe you still find deep-learning disappointing, but there have been some successes from having the computer learn multiple levels of internal structure/representation from data.

The problem is that many sentences (at least in English) exhibit some structural ambiguity (more than one valid syntax tree). Also you have to take into account pragmatics and sociolinguistic factors (like variation), and so on. You can't just feed a formal grammar to a program and expect it to correctly parse English. We're still a LONG way from being able to correctly parse all (or even 99%) of possible English sentences.

It does not depend on hard-coded rules. It learns various probabilities for sentence structures. The result is much more flexible than a formal grammar.

True, but that still means you require some decent priors about the text that someone reading it might have. People generally have an idea of what the person who wrote it might be like, what sort of things to disregard, and so on. My point was that learning the "structure" of the language is not enough. I don't know exactly how this program works though, so it might be decent at that, I haven't read up on it enough.

The program doesn't need to know priors, if you make sure the priors for the test set are close to the same as the ones for the training set :)

Deep learning is just a technique to do learning in (possibly many) layers.

And I disagree that trying to actually understand sentences is merely "slightly less shallow" than existing sentiment analysis systems (which, for the most part, treat sentences a just a bunch of words, count the positive and negative words, and take the average).

Reply on news.ycombinator.com