Hacker News Reader

Comment by saeranv | Hacker News Reader

I only recently started learning about vector embeddings, and the thing I can't wrap my head around is why cosine similarity should be so effective when (to my understanding) it should only capture linear similarity.

Shouldn't that "linear" constraint have more of a blunting effect on embeddings effectiveness? Why not use a different metric like mutual information that can account for non-linearities. Whenever correlation comes up in context of statistical comparison, there's always this warning that correlation is limited as a way of comparing data due to it's linearity so I'm confused why it seems so effective in the context of embeddings.

View on HN

The encoder is a neural network which is a nonlinear and highly expressive function family. The encoder function is optimized to produce a set of "factors" which are designed to be linearly comparable. If the "factors" aren't expressive enough, you can add more (increase the embedding dimension) or make them more sophisticated (increase the encoder network complexity). There's nothing stopping you from allowing them to interact in a more complex way as well, except that you give up the ability to easily do operations like indexing.

You've hit the nail on the head: the early Word2Vec models treat everything as a word, but words are polysemous and fluid. Mutual information is useful for discovering n-grams that can be informatively mapped to the same vector space as component 1-grams. (In fact, converting adjacent words with high pointwise mutual information such as "New" and "Zealand" to "New_Zealand" is an important preprocessing step noted by Mikolov, et al).

More recent vector embeddings (e.g., BERT, ELMO) are harder to conceptualize neatly because one of the layers of the Neural Network includes a token's indexical position in the document.

Mutual information relies on probability distributions, so I don’t think it makes sense to compare two random embeddings using it. Also, cosine similarities aren’t used in training BERT, I think it’s just cross entropy loss, which makes sense when doing classification.

Very good point. This is an active area of research, there's been some work in using other similarity measures like MI,Kendall's Tau etc in embeddings as well. And they have been found to perform pretty well.

I found something for MI if you wanna check it out : https://www.aclweb.org/anthology/2020.acl-main.741.pdf

EDIT: As @rubatuga said,they have been avoided largely as embeddings are continuous random vars and MI is for distributions. Nonetheless I think there can be merit in exploring these

PS: I am the author and just wanted to talk about the familiar basic stuff on this one

ha! no way - I just cited your paper in the comments without noticing you already had.

One of the most famous word embedding methods, Word2Vec, actually approximates word co-occurrence PMI matrix through matrix decomposition. Check out Omer Levy and Yoav Goldberg's work. In that case given enough dimensions it could approximate any matrix well - i.e. linearity shouldn't be a limiting factor.

Correlation (vs causation) and linearity are orthogonal concepts.

for all common models (GloVe, fastText, word2vec) the means across word embeddings are tightly concentrated around zero (relative to their dimensions), thus making the widely used cosine similarity practically equivalent to Pearson correlation https://www.aclweb.org/anthology/N19-1100/

Reply on news.ycombinator.com