Exploring Word2Vec
sujayskumar.blogspot.com
sujayskumar.blogspot.com
There is a very interesting new approach of learning Gaussian mixtures from words, which addresses this problem: https://github.com/benathi/word2gm
It also solves another interesting problem, which is that Word2Vec doesn't always put "categories" close to their actual contents.
Eg. I wrote a simple AI for playing Codenames with word vectors: https://github.com/thomasahle/codenames But where a human use the clue "Countries 2" for "Germany" and "England", the Glove vectors (at least in my implementation) much prefers the clue "France 2", which is pretty confusing at first.
However, in most cases it's probably actually better to ignore word order, since that captures more semantic relatedness rather than the syntactic features that influence word order. It also lets you translate between languages that do not necessarily have similar grammar: "Word Translation Without Parallel Data" https://arxiv.org/abs/1710.04087
Compare the following:
1) You must state the benefit.
2) You must benefit the state.
3) Feed fish vs. Fish feed.
Sometimes limits in inference:
1) A loves B.
2) Cats eat fish.
3) Cats are mammals.
Sometimes the sequence preceding or following will be important:
1) Problems with learning
2) Learning with problems
3) It has a learning problem
4) A learning problem for it.
Or consider an example from:
https://www.researchgate.net/publication/2335962_How_Well_Ca...
1) It was not the sales manager who hit the bottle that day, but the office worker with the serious drinking problem.
2) That day the office manager, who was drinking, hit the problem sales worker with the bottle, but it was not serious.
Order ends up mattering when you want to capture nuance or power higher order inference.
When you want to handle whole sentences, obviously order becomes much more relevant, but for that you'd feed the word vectors into an LSTM or similar to handle the order-dependence.
I will say that he’s wrong that NFL never co-occurs with ML. I’ve had discussions involving the No Free Lunch theorem by its initials.
Interestingly there's some controversy whether this benefit extends to punctuation. Most W2V tutorials instruct the programmer to apply tokenizing functions that leave only space-separated word characters, annihilating everything else, but I've actually observed higher accuracy scores from models that tokenize non-word characters individually, i.e. as strings of length 1. I'm not sure exactly why this is the case, but there is a common sense interpretation which might explain. Output from the precursory tokenizer output is often transduced by word2phrase, an optional bigram recognizer. Mikolov strongly recommends using it, but I'm sure it struggles to infer pointwise mutual information from a text that's missing ~70% of its contents.