Hacker News Reader

Comment by PaulHoule | Hacker News Reader

Has anyone developed commercial applications based on word embeddings?

It's clear that people are putting up better and better numbers on certain tasks that are widely shared, but for all I know these will always be a bridesmaid and never a bride when it comes to being useful for something.

Back in the 1970s it was clear that it wasn't going to be easy to make rule-based parsers that were "good enough" but it seems that now the task has been defined down so that if you can do better than chance that's a miracle. Thus people can kid themselves into thinking they are practicing what Thomas Kuhn called "normal science" since they are in the same shared reality even if it is a delusion.

View on HN

I train NER and text classification models using Spacy. You can give Spacy word vectors and the accuracy usually increases 1-5% in my experience. I use NER to classify entities within search keywords, text classification for determining if a new court docket is relevant to a research group I work for, and I have started working on an ML-assisted structured data extraction from free text tool. All of these things are part of commercial products or will be.

what is the before and after accuracy? how big of a training set?

Probably easiest to talk about the NER for search keywords. I have found 1000-2000 examples per label is enough to get >80% accuracy. I don't remember the exact before/after adding pre-trained word vectors, but it was a 2-3% improvement and the current F1 score is ~86% I believe.

I have found it best to apply this to search tail queries (uncommon queries) because the relevance already sucks so any improvement is helpful. Often they are misspellings or unique wordings of common queries. If you have a product search engine, and you can recognize the product category then you have a much better chance of giving the user a relevant result even if the words are misspelled. If you can improve the overall relevance of tail queries, then revenue for the product search engine should increase as well.

I should also add that Spacy turns all words into vectors under-the-hood. Providing pre-trained word vectors accelerates the training process so you don't need as many examples (hopefully).

Most modern NLP methods are built on top of word embeddings, things like Neural Machine Translation, text classification and etc all convert input words into word embeddings then stack a neural networks on top of it.

People write papers about these things.

"Do they make commercial applications?" is the question.

The paradigm here is "Does method X get better answers than method Y?" as opposed to "Will we embarass ourselves if we put method Z into production?"

All of those things are used daily by billions of people around the world. Google sells their translation system through Google cloud, so does AWS and a bunch of other NLP startups.

Can you point me to a text analysis API that is not embarrassingly bad?

I have tried products from Amazon, IBM, Google and other companies and the one thing they have in common is they've never passed an acceptance test for me.

Language translation is a particular bad example of something where a "Clever Hans" effect can happen, where the reader's desire and capability for closure will fill in for mistakes the system makes and make it look like it performs better than it really does.

I don’t use off the shelf APIs, but I’d be surprised if Google’s named entity recogniser isn’t pretty good in the news or financial domain.

OpenCalais is pretty good in the financial domain. http://www.opencalais.com/opencalais-demo/

IBM isn’t great. Don’t know about Amazon.

That's true, but specifically for translation I think all SOTA methods use task-specific embeddings (i.e., learn their own instead of using BERT or Elmo, etc.). See https://arxiv.org/pdf/1808.09381.pdf and note that the BERT paper doesn't mention translation as a metric.

I build commercial applications built on the FastAI language model (similar to this) in a legal domain.

It’s a per sentence matching task, and it increased the accuracy ~5% (!!) over a tuned FastText classifier, which in turn was much better than a traditional n-gram based classifier.

The accuracy is now good enough (and human like enough) that trained humans generally disagree with each other when trying to correct its “errors”.

Training set size was ~1000 labeled examples.

I work at a company that uses word embeddings to generate predictions that customers actually pay for. There are lots of other things going on under the hood (e.g. neural networks in which the embeddings are the first layer, non-ML software architecture to make the UX something worth paying for) but to answer your question, embeddings are definitely generating real value.

Reply on news.ycombinator.com