Hacker News Reader

Comment by minimaxir | Hacker News Reader

Word2Vec and bag-of-words/tf-idf are somewhat obsolete in 2018 for modeling. For classification tasks, fasttext (https://github.com/facebookresearch/fastText) performs better and faster.

Fasttext is also available in the popular NLP Python library gensim, with a good demo notebook: https://radimrehurek.com/gensim/models/fasttext.html

And of course, if you have a GPU, recurrent neural networks (or other deep learning architectures) are the endgame for the remaining 10% of problems (a good example is SpaCy's DL implementation: https://spacy.io/). Or use those libraries to incorporate fasttext for text encoding, which has worked well in my use cases.

View on HN

Author here. One of the reasons to start with something even simpler than embeddings such as fastText/Word2Vec, is their ease of explainability. As this blog post alludes to, the majority of ML work is data work, and in order to efficiently work on your data, understanding what your model does right/wrong is critical. This is why we favor starting with approaches that might seem simple or outdated. Once you see how these approaches fail, you can better inform your choice of a complex model.

Having worked on modeling data and sharing results with customers, I agree simpler solutions can be better for ease of explaining alone. Additionally, the loss of x percent accuracy with lower-tech solutions can sometimes be worth it because they are easier to train and reason with. This is particularly the case when you are just looking for loose directional indicators and correlations.

If simple solution gets 77% accuracy, and complex solution yields 80%, I would wager that most of the time you should just stick with the simple solution.

One specific example that comes to mind is in sentiment analysis where you can achieve "sufficiently" high accuracy with well tailored Bayesian approaches. They are super fast to train and reason with. If a customer/consumer wants to know exactly why a piece of text is positive or negative, the n-gram probability matrix is extremely easy to inspect. Subsequent re-training and fixing is also much easier than re-training a large neural network, svm, etc.

Can you explain how you reduced sentences to 2 dimensional entities using PCA ?

First, FastText is very similar to (a superset of) Word2Vec. Secondly. RNNs are far from being endgame for anything. In fact, recent research has been moving away from them towards more predictable and parallelizable methods (see, e.g., Google's Transformers, aka "Attention is all you need", or FAIR's approach to translation using CNNs).

Added a note about other DL architectures.

You may be painting with too broad a brush. I've worked with a number of NLP people to try to beat our tfidf model using deep methods and various word vectors. The tfidf still performs better. I think the reasons are: many out-of-vocab words, and small amounts of labelled data.

Coming a bit late to the discussion, but out-of-vocab words and small amounts of labelled data is the exact kind of problem that word embedding vectors are meant to solve, and they do so exceedingly well. Of course, you need a good, large source of (unlabelled) data, preferably approaching the gold standard of "everything that has ever been written about your problem domain" to train the embeddings, and that takes some effort, but then the out-of-vocab words must be exceedingly rare (i.e. only if it's a genuinely new word that has never ever been mentioned before, or a very very rare term that you've discarded), and even that can be solved well with character-based embeddings.

I built a docker image exposing spacy.io (with their largest en model) as rest endpoints

http://github.com/artpar/languagecrunch

I'm not an expert but I'd be surprised if you could declare any approach "endgame" at this point.

My interpretation is that they meant something like "last resort" there.

Fasttext is not that different from word2vec

Fasttext's ability to handle OOVs, however naïve, cannot be understated. I'm not sure about the literature on this, but I think generalizing based on character-gram wordvectors is probably better than assigning a dropout based common OOV vector.

Evolution for sure, heck, they even has the same author. But fasttext is the same idea as word2vec, the objective is identical.

Reading the guide for fasttext I am not sure how a non NLP researcher would use it.

"This library can also be used to train supervised text classifiers" - https://github.com/facebookresearch/fastText#text-classifica...

Looks like you can use it for gensim. https://radimrehurek.com/gensim/models/fasttext.html

You could probably throw DNA, RNA or protein sequences at it.

A lot of people do. Here are the academic articles that cite Gensim and contain the word "protein" (4 pages of them):

https://scholar.google.com/scholar?hl=en&as_sdt=2005&cites=1...

I think you'd be surprised to find out that for a pretty large amount of classification problems, tf idf and Bayes naive classifiers do equally as good or almost as good but with less variance than deep nets.

also afaik fastext is just a faster way to build embeddings that are mostly equivalent to word2vec

All and all, I'm not sure you're enough of an expert to be commenting on this

1) No, they don’t. Eg, on the current Toxic comment classification TF-IDF and Bayes models are about 20% worse than RNN models.

2) FastText is also a classification algorithm, a command line classifier and a set of pretrained embeddings.

3) Yes, he does. I’ve learnt from his notebooks before, and I am confident I’m qualified to judge. I don’t completely agree with his statement here, but he is well qualified to comment.

1) I'm saying, depends on the size of the datasets

one of fasttext's main features is a pre-trained word2vec implementation, you can hardly call word2vec obsolete. Bag of words has been obsolete for a long time of course.

Yeah, fasttext/spacy/gensim are some of the biggest open source NLP libraries these days. However I've found that they aren't very performant - I've done prototyping in them before but wouldn't use them for a finished product. I'm sure most big tech companies use custom implementations to do their heavy lifting

It's hard to quantify "most", but I can attest that some of the largest companies in the world do in fact use Gensim, in part because of its good performance and robust battle-tested implementations.

I'm honored that you would respond to my comment! I meant that in no way as a dig against Gensim. Really I'm referring more to the pythonic environment itself - the larger an NLP application grows within Python, the more overhead baggage it picks up (due to logic written in Python - plus Cython is far from a silver bullet). That is not to mention all the difficulties involved in parallelization within a Python environment.

For what it's worth, when I was at a large company recently, I did use Gensim too - I was originally referring more to the case where you take a prototype for an NLP application and ask "ok, how do we scale this across millions of users"

The Gensim version of FastText doesn’t support subword embeddings without additional support.

FastText itself (the command line program and the pretrained embeddings together) is a decent starting point for classification though.

The latest gensim does support fasttext .bin files and subwords (certainly using it with out of vocabulary words right now.) It doesn't support fasttext classification though.

Oh that's awesome! I've been pre-processing and generating separate embedding files.

Bag-of-words/tf-idf is still a pretty good goto for working out "significant terms" to solve "search" tasks. Its simple, old, and reasonably effective, which is what makes it good.

Fastext is faster but does not beat vdcnn and other deep architectures. It all depends on the use case really.

When you say GPU + RNNs address the remaining 10% of problems... what problems fall under that 10%?

In my case, problems where text is not the only input into the model (but the text is a highly-important feature), the text document is only a couple characters/words (or empty), and/or regressions instead of classifications. (although most of those are not specific to text encoding, but are difficult to implement and get good results without tools like Keras regardless)

Reply on news.ycombinator.com