Comment by wenc | Hacker News Reader

Comment by wenc | Hacker News Reader

Parent Full thread

The cognoscenti already know this, but word vectors are a game changer in the multi-class text classification game. Finding the correct representation of text makes the classification task so much easier.

For my problem (1000+ total classes, 1 class per input), I experimented with Naive Bayes + TFIDF (~50% accuracy, < 1 sec training), then Word2vec + CNN model on GPU (~70% accuracy, 6 hrs training), and finally FastText (99% accuracy, 10 minutes training).

FastText [0] in particular is quite impressive. It is essentially a variant of Word2Vec that also supports n-grams, and there is a default implementation in C++ with a built-in classifier that runs on the command-line (no need to setup Tensorflow or anything like that).

Despite it only running on plain CPUs and only supporting a linear classifier, it seems to beat GPU-trained Word2Vec CNN models in both accuracy and speed in my use cases. I later discovered this paper from the authors comparing CNNs (and other algorithms) to FastText, and their results track my experiences [1].

This goes to show that while GPU-accelerated models are cool, sometimes using a simpler, more suitable model can have a significantly better pay-off.

[0] https://fasttext.cc/docs/en/supervised-tutorial.html

[1] https://arxiv.org/abs/1607.01759

Thanks for this, I'll have to take a look at FastText. I've been using word2vec before turning it into a matrix and running it through a CNN. I based it off of Yoon Kim's[0] work. I haven't had much luck though on my 92-class problem. Maybe Fast Text will work better, although I think there are a lot of improvements my model can have still.

[0] https://arxiv.org/abs/1408.5882

I'm working on a problem kinda similar to that with a binary classification on a class imbalanced dataset, but fastText and Bidirectional LSTMs appear to work pretty terribly even with oversampling. Is there a better alternative?

Class imbalance is a tricky problem, but it is unrelated to FastText.

There is no silver bullet. The best solution is to collect more data to bring the classes to a balance. The second best approach is to try algorithms like SMOTE.

SMOTE is for more numerical data.

It is a numerical problem underneath. Word vectors are numerical vectors.

>FastText [0] in particular is quite impressive. It is essentially a variant of Word2Vec that also supports n-grams,

To clarify: fasttext does not support n-grams of words, but instead considers n-grams of characters within words.

It supports word n-grams as well as character n-grams https://fasttext.cc/docs/en/options.html

See options -minn, -maxn and -wordNgrams.

If the word embedding is, say, 400 dimensions, are you averaging the vectors of all the words in the document to get a document vector?

For the CNN model I was using Magpie [0] which takes a sum of the vectors (so yes, equivalent to unweighted average) to represent the document.

For the FastText model, it would be whatever FastText is doing under the hood. I haven't peeked at the code to see what it is doing, but Section 2 of the paper I cited above seems to imply that an average is taken.

[0] https://github.com/inspirehep/magpie

You're correct! The keras example implementation of FastText uses a global average pooling is used to reduce the dimensionality [0].

[0] https://github.com/keras-team/keras/blob/master/examples/imd...

Interesting. I had heard specifically that word vectors weren't a game-changer for document classification, because the averaging method didn't work well. But the difference between word2vec and FastText must be important. I think your baseline of Naive Bayes + TF-IDF would do a lot better if it was SVM and TF-IDF, but I didn't expect the big jump between word2vec and FastText.

> I had heard specifically that word vectors weren't a game-changer for document classification, because the averaging method didn't work well.

As with anything, your mileage may vary.

One aspect of FastText that definitely helped in my case was n-gram support (both word and character, tunable via command-line arguments). In my corpus, I have short fragments of sentences containing misspelled words, incorrect grammar etc. plus my test set has out-of-vocabulary words.

n-grams are more robust to these than Word2Vec which uses a static vocabulary.

I thanked your top level comment, but this whole thread is great. I fond my self worth data that sounds like yours. I'm excited to try fast text.

I think it's important to use the right "average" operation. word2vec puts words on the surface of a hypersphere, so "adding" vectors means turning around the sphere.

Just piping in to say that I'm very curious as well! How do you go from word vectors to a description of the document?

Combining word embeddings to get a single representation of a sentence in general is a pretty hard problem. Point-wise averaging, addition or summation [0] works alright, but it's not great. Some more recent work looks a using RNNs to combine them; most of the literature I've seen is pretty application-specific, but try [1][2] for flavour.

[0] http://www.aclweb.org/anthology/P/P08/P08-1028

[1] http://arxiv.org/abs/1409.3215

[2] http://arxiv.org/abs/1409.0473

[0] link should be http://www.aclweb.org/anthology/P/P08/P08-1028.pdf Jeff Mitchell; Mirella Lapata - Vector-based Models of Semantic Composition

Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors.

Other approaches like doc2vec are also used.

Thank you for this very helpful information, and I have just one more question:

"Often the w2v embedding layer will be the first layer of a network, through which a document of word representations will be passed. The output of the embedding layer will be a 2 dimensional tensor with the embedding dimension in one direction and the number of input words in the other. The next step is often to apply zero or more convolutional units over the word direction before some kind of pooling to get a 1 dimensional output. The output of the pooling layer will be a document representation based upon the word vectors. Other approaches like doc2vec are also used."

I thought the approach you described was doc2vec. If not, then does it have a name/citation?

Doc2vec is the name of the gensim implementation of this paper [0]. Briefly, it creates document embeddings (document being anything on the level of phrases, to sentences, to paragraphs and beyond) on the basis of either predicting a word given the context words (distributed memory -- DM), or predicting the context words given a word (distributed bag of words -- DBOW).

[0] https://arxiv.org/abs/1405.4053

So in simple terms, you concatenate the word vectors and feed them to further network layers to arrive at a reduced-D representation?

Either way... Very nice, thanks!

The simplest thing, and one that apparently works fairly well with fasttext vectors, is just to average them.

I'm surprised (and delighted) you say that such a straightforward approach works well, thanks.

Reply on news.ycombinator.com