Ignorant question about the algorithm:
There is a min_frequency threshold for including words
Wouldn’t you want a max_frequency instead?
A lot of times infrequent words are more important than the frequent ones (eg titles vs articles)
There is a min_frequency threshold for including words
Wouldn’t you want a max_frequency instead?
A lot of times infrequent words are more important than the frequent ones (eg titles vs articles)
It is also worth discussing the difference between the training corpus (the data that the word vectors are found from) and the inference corpus (the data that the word vectors are used on). If a word is uncommon in the inference corpus, but is common enough to be trained on the training corpus, then a vector is still generated and useful for the inference task.
Is the context considered “all appearances of the word”, or “text around the word”?
Also, are all positions/“areas” of the corpus considered evenly important except for their frequency?
In other words, ie position within the corpus considered at all? (eg. Usually titles are more relevant to a document then a random sentence or word in the middle of it)
For example in the sentence:
> Usually titles are more relevant to a document then a random sentence or word in the middle of it
If 'document' is the current target word, we select ["relevant", "to", "a"] and ["then, "a", "random"] as our left and right contexts. These context words and our target word, ["document"] form our 'positive' examples that we want to move closer together in the embedding space.
Position in the text is not used in this algorithm. You're right about titles being more relevant to the meaning of a document but the document isn't important: it's just a resource that we're using to learn which words occur together (and thus are semantically related). The algorithm steps through every single word as a target and thus will inevitably utilize words that were titles just as much as words in any other part of the text:)
Hope that helps!
Yes, very much
Thank you for such a great explanation, super clear :)