Is the context considered “all appearances of the word”, or “text around the word”?
Also, are all positions/“areas” of the corpus considered evenly important except for their frequency?
In other words, ie position within the corpus considered at all? (eg. Usually titles are more relevant to a document then a random sentence or word in the middle of it)
For example in the sentence:
> Usually titles are more relevant to a document then a random sentence or word in the middle of it
If 'document' is the current target word, we select ["relevant", "to", "a"] and ["then, "a", "random"] as our left and right contexts. These context words and our target word, ["document"] form our 'positive' examples that we want to move closer together in the embedding space.
Position in the text is not used in this algorithm. You're right about titles being more relevant to the meaning of a document but the document isn't important: it's just a resource that we're using to learn which words occur together (and thus are semantically related). The algorithm steps through every single word as a target and thus will inevitably utilize words that were titles just as much as words in any other part of the text:)
Hope that helps!
Yes, very much
Thank you for such a great explanation, super clear :)