Google Is Working on a New Type of Algorithm | Hacker News Reader

Google Is Working on a New Type of Algorithm

wtvox.com

This is not about word2vec, an algorithm for learning vector representations of words.

This is about learning vector representations of thoughts (ideas, concepts, etc.) in an unsupervised manner.

For reference, two months ago some of Hinton's colleagues at the University of Toronto published a paper describing an unsupervised algorithm for learning vector representations of sentences: http://arxiv.org/pdf/1506.06726v1.pdf

Once trained, this algorithm is given a sentence and asked to find similar sentences in a corpus of half a million sentences, and it produces impressive results. For example, when asked to find the sentence with most similar vector representation of sentence 1, it finds sentence 2:

* Sentence 1 (sentence given to the trained algorithm): "He ran his hand inside his coat, double-checking that the unopened letter was still there."

* Sentence 2 (sentence with most similar vector representation found by algorithm in corpus): "He slipped his hand between his coat and his shirt, where the folded copies lay in a brown envelope."

The paper has more examples. The vector representations learned by this type of algorithm truly appear to be capturing the meaning of sentences.

This looks promising.

Edit: Code used in the paper I mention is/will be here: https://github.com/ryankiros/skip-thoughts

> This is not about word2vec

Thanks for the clarification. I actually thought they were talking about word2vec.

The "Paris - France + Italy = Rome" example is really similar to other word2vec examples.

How is this different from building a huge database of n-grams and doing probabilistic pattern matching on them?

More info (references, source code, data sets) at https://code.google.com/p/word2vec/

Edit: I couldn't figure that out from the text, but according to a comment by cs702, this isn't about word2vec, but to me, it seems 'just' a more advanced version of it.

Previously: https://news.ycombinator.com/item?id=10100910

>"If you take the vector for Paris and subtract the vector for France and add Italy, you get Rome"

I would have liked to hear more about the parts that are not purely based on associations between words given some context. Associations only get us so far. Eventually, we will need to extract logic as well.

Very interesting. It is based on a technology of semantics or is it a neural network where each neuron represents a concept?

It sounds like this is something like Word2Vec on steroids, i.e., some kind of dimensional reduction based on seeing lots of instances.

It is an interesting idea but my suspicion is that it is a way to get where current NLP technology is with less work but it won't quite "cross the chasm" to produce commercially useful systems.

Deep learning is an academic bubble that is going to sweep through the problems that is good at, but 20 years from now people are going to look back it like many look at symbolic AI, which swept through the problems it was good at and then ran out of steam.

Pity the article is so frustrating to read.

http://i.imgur.com/youRjpv.png

Reply on news.ycombinator.com