Twitter Natural Language Processing | Hacker News Reader

Twitter Natural Language Processing

ark.cs.cmu.edu

207 pointsjjwiseman29 comments

For those that didn't click though, the tweets they show for their POS tagger are great:

   ikr smh he asked fir yo last name so he can add u on fb lololol

and

   :o :/ :'( >:o (: :) >.< XD -__- o.O ;D :-) @_@ :P 8D :1 >:( :D =| ") :> ....

I'd love to see what Word2Vec did for that first one.

POS is an unfortunate acronym. I genuinely thought you were being derogatory and therefore sarcastic at first! Thus, the challenge of natural-language processing; I wonder if the tagger is actually more intelligent than I am ...

And to me I thought "point of sale" rather than "piece of shit", lots of options!

The demonstrates the importance of being able to parse languages other than english.

I'm scared. Does this mean there's some value to parsing this kind of gibberish? If so, does the fact that I consider it gibberish mean I'm going to miss the bus on future developments because I've decided I'm too old to take people who write like this seriously?

Until someone starts parsing it, how would we know if there is value or not?

CMU does some of the most interesting work around with data from Twitter. Some of the same people responsible for this worked on predicting NFL games with Twitter data[0]

[0]https://www.cs.cmu.edu/~nasmith/papers/sinha+dyer+gimpel+smi...

Yes, Noah Smith's NLP group at CMU is awesome. Btw, Noah is moving to UWash.

Tweet NLP from CMU.

  We provide a tokenizer, a part-of-speech tagger,
  hierarchical word clusters, and a dependency parser for
  tweets, along with annotated corpora and web-based
  annotation tools.

No freaking way.

Literally 2 hours ago I was in professor Kosbie's office talking about a CS term project I want to do involving Twitter and NLP...

were you in the room??

i doubt that it's so much of a rare occurrence. NLP and machine learning are pretty much popular right not.

that like having 2 persons with the same first name in a group of 40 people or more.

Birthday paradox!

Maybe Professor Kosbie has read all those papers ... the TwitterNLP tools were published between 2011 and 2013.

I wonder why Peen Treebank POS tags are not the default, instead of some new custom tag syntax.

The default tagset that this employs is much more coarse-grained than the Penn tagset. For example, it has only two verb tags, whereas Penn has separate tags for infinitives, gerunds, past forms, etc. On the other hand, it supports the peculiar kinds of lexemes that abound in tweets, having separate tags for hashtags, @-mentions, URLs, emoticons, etc.

Twitterese English is very different from standard English, and by taking the dedicated-tagset approach that embraces this fact, the authors were able to accurately attribute structure even to text that appears gibberish at first sight. To me, this is very impressive.

Part of it is genuine differences between online conversational language versus standard written English, like emoticons, Twitter-specific discourse markers, and hard-to-segment compounds or clitic constructions (see the Gimpel and Owoputi papers (2011, 2013) linked on the page, and/or the annotation guidelines document too). Part of it is just that it's easier for humans to annotate the coarse-grained POS tagset, and we didn't have many resources for annotation when we did it.

These things also intersect ... for example, you'd have to figure out how dialectical English verbal auxiliaries like "finna", or the second or so word in "imma", map to PTB tags. It's possible but just takes more work and thinking through the descriptive linguistics and what you want to use it for. Someday I'd like to update the whole thing for a more PTB-like POS tagset, if it can be done well. I feel like Chris Manning's whitepaper on issues in PTB POS data convinced us (well, it convinced me, at least) that it might be a good idea to focus on making high quality tag annotations. (http://nlp.stanford.edu/pubs/CICLing2011-manning-tagging.pdf )

People at UWash (Ritter et al.) have Twitter tools with a tagset closer to PTB tagset (they still have special tags for @mentions and #hashtags)

Yeah, that's the main PTB-style tagset resource for tweets (that I know of). Our tool can also be run to produce tags in their form (we just retrained our software on their annotated data). See further down the webpage for information on how to get this.

Very cool. I did some authorship attribution using a twitter corpus from MSR during grad school. The size constraint for a tweet causes users to have a different writing style then on other document types, which makes for interesting NLP problems.

Great, I've been waiting for the release of TweeboParser. Just before EMNLP14 gets started.

GPL :( can't use it

Why not? Contact the author(s) and ask if they'll give you non-GPL license.

The whole project is based on various libraries. In particular, the POS tagger itself uses the OWLQN optimizer from Stanford NLP (licensed under GPL).

However, it's possible to remove GPL libraries out of the POS tagger as mentioned here: https://github.com/brendano/ark-tweet-nlp/blob/master/LICENS...

You may want to look at Factorie (https://github.com/factorie/factorie), that has a decent POS tagger and it's not crippled by the license. It also has dependency parsing which works reasonably well.

I've been looking at Factorie for a while but haven't actually done anything heavy with it.

I planned to replace the optimizer in CMU's POS tagger with the one implemented in OpenNLP to make the tagger fully Apache. Unfortunately, so busy right now. Currently, I'm running the tagger on AWS, so the GPL doesn't hurt me much.

BTW, besides the POS tagger, CMU's TweeboParser depends on Turbo Parser which again is licensed under GPL.

it's not as simple as that, the whole stack is GPL, mainly due to Standford's usage of GPL for all of their NLP.

GPL is a killer for any project that's not a toy or for school.

It depends how the project ownership and copyright is structures. We have used GPL projects successfully when they are dual licensed. It never hurts to ask anyway.

GPL is a killer for any project that's not a toy or for school.

My paycheck - financed by some of the largest companies in my country - disagrees. There are other ways to make money besides locking others from your code.

Dr. O'Connor teaches my NLP class. His work is revolutionary.

Reply on news.ycombinator.com