What Ive wondered about word-vectors is, are different meanings split or not?
A word having many meanings would otherwise means its vectors are more "smeared" out and less accurate I guess. Although, I guess that might be good for these algorithms.
A word having many meanings would otherwise means its vectors are more "smeared" out and less accurate I guess. Although, I guess that might be good for these algorithms.
In practice, as you intuit, contextual embeddings are in fact useful for obtaining more precise word vectors. The example given in[0] is BERT's embeddings, though I would imagine that most NLP applications nowadays all use contextual embeddings since they're a strict improvement over static embeddings.
[0]. https://web.stanford.edu/~jurafsky/slp3/6.pdf (search for 'static embeddings')
So in the scenario that these are products which have IDs, and the container is a document that mentions these products together, you can reduce it to a key-value map of documentID-[list of product IDs], and then by selecting the vectors that have cosine similarity of .95-0.9999, you can get the object similar to that product but not that product. Using contextualized vectors, in the best case would give you things similar as well as the query product, but for every single instance, not just the single abstract entity.
With a high dimension vector (>50d, more usually 300d or 500d) one or two dimensions together will pick up a concept.
This is why word algebra like
Paris - France + Russia ~ Moscow
works with original Word2Vec vectors.
Language models are better at picking up context-dependent meaning, but word vectors still have good enough performance for many tasks.