Yes, that’s what the glove paper showed. Linear substructure in PCA space.
does that apply for ada-002 embeddings, if you don't use GloVe? I would think it only applies if you create embeddings using a linear model?
I'm not a mathematician but on their site it says - "Cosine similarity and Euclidean distance will result in the identical rankings" so I am assuming that would have to be the case?
When you have unit vectors, cosine similarity and euclidean distance always result in identical rankings, even when those unit vectors are assigned at random and have no semantic structure at all. That's because the euclidean distance |a - b|² = ⟨a - b, a - b⟩ = ⟨a, a⟩ - 2⟨a, b⟩ + ⟨b, b⟩ = |a|² - 2|a||b|cos ɸ + |b|² for unit vectors with |a| = |b| = 1 only depends on the cosine of the angle ɸ between the two vectors.
Pythagoras strikes again.
Not always, another comment mentions it is true for GloVe, but that doesn't mean it is true for all model's vector spaces.
In a "good" embedding, they do.
When I was first reading about Word2vec I thought it was absolutely wonderful that the vector delta (king - queen) was similar to the vector delta (man - woman). That captures relationships in such a fascinating way!
It is not obvious to me why word2vec's training objective yields this. word2vec ensures similarity of related words, but why can you then perform linear algebra on unrelated words?
The training objective does not have much to do with it, the bigger reason is that these neural networks themselves use linear algebra (dot products/projections and whatnot) to manipulate embeddings.