Hacker News Reader

Comment by garysieling | Hacker News Reader

For integrating into Solr, I've used Word2vec to improve rankings the synonyms it finds (boosting synonyms by how similar they are to the query). In English Word2vec tends to think plurals are synonyms.

I did a talk on this with more details: https://www.slideshare.net/GarySieling/ai-with-the-best-buil...

Sense2Vec might also help: https://github.com/explosion/sense2vec

There is also a project called Vespa, which looks potentially interesting as a replacement for Lucene - https://github.com/vespa-engine/vespa

View on HN

The dice talk mentions that also (weighting by word2vec similarity). It's important to note that word2vec, LSA, sense2vec, etc, all find words that are RELATED but not necessarily SYNONYMOUS. For instance, antonyms like black and white, rich and poor, often appear in the same contexts, have the same word type but are opposite in meaning. Similarly, politicians on the opposite ends of the political spectrum will usually get assigned similar vectors and the same cluster as they tend to appear in similar contexts. It uses the context (word window in word2vec and GloVe, the document in LSA) to determine a measure of similarity. But the context for antonyms is typically very similar. Attaching the part-of-speech tag to each word before pushing it through these models can help as it enforces that grammatical relation, but this won't address all of these issues (e.g. black and white are both adjectives). In my experience, if people mostly search for nouns in your search engine (e.g. job search) this issue is also less of a concern, but can still cause cause problems. Finally, conceptual search can also help with precision - by matching across all concepts within a document, you can help disambiguate its meaning, when you have words that have multiple meanings.

> Finally, conceptual search can also help with precision - by matching across all concepts within a document, you can help disambiguate its meaning, when you have words that have multiple meanings

Thanks! I've written up something along these lines here: https://news.ycombinator.com/item?id=15592196

I'd love to hear your opinion if this is going to work.

Reply on news.ycombinator.com