Python NLTK Bayesian Classifier for word sense disambiguation - 92% accuracy
litfuel.net
litfuel.net
Bag-of-words models perform pretty well at classification and search, and the main thing you need to improve search is to boost scores when words are close together.
You might think you could improve performance by using semantically better defined features, but even 92% accuracy adds enough noise to foil your plans.
It's a big problem in A.I. systems that have multiple stages. You might have 5 steps in a chain which are each 90% accurate, but put them together and you've got a system that sucks. Ultimately there's a need for a holistic approach that can use higher-level information to fix mistakes and ambiguities at the lower levels.
Many areas in NLP are like this. You can get 92% accuracy in a few hours of work, and then you can get 93% after a week or work, and then you can write a whole PhD thesis about how you got 94% accuracy.
To a certain extent, there are approaches, such as the Support Vector Machine that are "unreasonably effective" but once you get past that, you often have to confront issues that everybody wants to sweep under the rug to make a real breakthrough.
For instance, there was that NELL paper that came out a few months ago; NELL extracted facts from text but it had no idea that "Barack Obama is the President of the United States" was true in 2010, and that "Richard Nixon is the President of the United States" was true in 1972. If you can't handle the fact that different people believe different things and that statements have expiration dates, no wonder you can only get 70% accuracy in IX
That said, text-to-speech is a system where it's important to do disambiguation of a particular set of words. For instead,
"I read the news today, oh boy", "read" sounds like "red"
"I read the news every day", "read" sounds like "reed"
You need to be able to disambiguate the word sense to be able to correctly read the world "read". There are maybe 20 or so very common words that are like this, so a modest amount of work in this area would be part of a good TTS system.
Here are some questions:
- What happens when we change the language model? - What happens when we intersperse language models (English phrases within Chinese)? - What if someone were to just say "i love apple"?
This post title is also very misleading. The 92% accuracy reflects only one particular use case. How about attempting to disambiguate hundreds and thousands of terms?
I'm quite interested in how will you approach this problem ?
Has anyone else seen this elsewhere? It's new to me and I was surprised by how obnoxious it was given that the web isn't exactly a stranger to obnoxious flashing content.
Interesting. I asked a few coworkers if it was just me and they confirmed it. The actual favicon doesn't blink for you? http://www.litfuel.net/favicon.ico
It reports itself as a 6 frame gif for me. Maybe your browser is just more sane than mine (Firefox 3.x) and refuses to honor animated favicons?
I figured I was getting rightly downvoted because I wasn't saying anything about NLTK.
In my local version of firefox, the entire icon vanishes for about 5 seconds, every 5 seconds. Causing a notable visual disturbance. In Chrome on another system the eyes blink every 5 seconds or so. So yeah, my browser is being broken.
But when I asked people if it blinked ... they, of course, said yes.
EDIT: Also, as noted in an edit above, when viewed in Chrome it didn't animate unless the image was accessed directly. So for you it may blink, not blink, or really blink, depending on your browser and configuration...
- The article title implies that this is somehow a spectacular finding. Doing word sense disambiguation for one word is not that interesting, and there is no comparison with existing methods to show that this is actually a high score. I suspect that it is not that spectacular, since 'Apple' is relatively easy to disambiguate using a few context words.
[1] E.g. see:
- Using Wikipedia for Automatic Word Sense Disambiguation, R. Mihalcea, 2007, for a discussion of using Wikipedia to train a word sense disambiguator.
- Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach, H.T. Ng and H.B. Lee, 1996, provide a good overview of types of features that can be used in disambiguation. They use features that go beyond simple 'bag of word' and 'bag of n-gram' features, e.g. by using syntactical patterns.
There is a whole lot more research of course, but just to show two examples that describe far more sophisticated approaches.
It is a testmanet to NLTK that this can be accomplished in less than 100 lines.
- In naive Bayes classification, model parameters can usually be estimated using relative frequencies in the training data.
- WordPunctTokenizer is a very simple tokenizer that makes anything matching \w+ and [^\w\s]+ a separate token.
- Extracting Bigrams from a list of tokens is trivial.
Of course, using NLTK will be very helpful in many situations, but this is hardly a testament to NLTK.
It's still fun to remember how quick and easy something like this is though. Any interest in similar articles on named entity recognition, sentiment/topic classification and spam filtering? I've been meaning to do a few for a while, but you know how it is.
GBs of data are pretty easy to handle these days