Comment by polm23 | Hacker News Reader

Comment by polm23 | Hacker News Reader

This is an interesting surprise about good old word vectors. From the README:

> Although continuous bag of word (CBOW) embeddings can be trained more quickly than skipgram (SG) embeddings, it is a common belief that SG embeddings tend to perform better in practice. This was observed by the original authors of Word2Vec [1] and also in subsequent work [2]. However, we found that popular implementations of word2vec with negative sampling such as word2vec and gensim do not implement the CBOW update correctly, thus potentially leading to misconceptions about the performance of CBOW embeddings when trained correctly.

The upshot is that they get similar results with CBOW while training three times faster than skipgram.

Given the popularity of Transformers, and that Fasttext exists, I'm curious as to what inspired them to even try this, but it's certainly an interesting result. There's so much word vector research that relies on quirks of the word2vec implementation.

I'm not surprised that industry prefers W2V over Transformers, given how heavy-duty the latter can be at inference time.

It's been a few years since I looked at it, but IIRC fastText is basically just w2v with subwords, so it's also possible this negative sampling fix applies to w2v and fastText equally.

One can also use shallow transformer models if inference throughput is important.

With all the eyeballs on word2vec and gensim how did this not get picked up before?

Gensim considers the `word2vec.c` code, from the original authors of the Word2Vec paper, as canonical and seeks to match its behavior exactly, even in ways it might deviate from some interpretations of the paper.

If there's an actual benefit to be had here, Gensim could add it as an option - but would likely always default to the same CBOW behavior as in `word2vec.c` (& similarly, FastText) - rather than this 'koan' variant.

Apparently it did: https://github.com/RaRe-Technologies/gensim/issues/1873

While I still need to read this paper in detail, I'm not sure their only change is to this scaling of the update.

The `koan` CBOW change has mixed effects on benchmarks, and makes their implementation no longer match the choices of the original, canonical `word2vec.c` release from the original Google authors of the word2vec paper. (Or, by my understanding, the CBOW mode of the FastText code.)

So all the reasoning in that issue for why Gensim didn't want to make any change stands. Of course, if there's an alternate mode that offers proven benefits, it'd be a welcome suggestion/addition. (At this point, it's possible that simply using the `cbow_mean=0` sum-rather-than-average mode, or a different starting `alpha`, matches any claimed benefits of koan_CBOW.)

The paper itself says the only change is normalizing by the context window size C.

Ah, but I've now looked at their code, and it's not the only change! They've also eliminated the `reduced_window` method of weighting-by-distance that's present in `word2vec.c`, Gensim, and FastText.

What if that's the real reason for their sometimes slightly-better, sometimes slightly-worse performance on some benchmarks? Perhaps there are other changes, too.

This is why I continue to think Gensim's policy of matching the reference implementations from the original authors, at least by default, is usually the best policy – rather than using an alternate interpretation of the often-underspecified papers.

The word2vec implementation has many details that are unmentioned or at least not emphasized much in the paper. The source is also not very commented if memory serves.

This is another paper that's basically just about some details of word2vec and GloVe and their effects on the results:

Improving Distributional Similarity with Lessons Learned from Word Embeddings - ACL Anthology https://www.aclweb.org/anthology/Q15-1016/

To save a few clicks, here's the paper that describes the fix and gives some comparisons with the supposedly broken implementation.

https://arxiv.org/pdf/2012.15332.pdf

While I'm unsure of this paper/implementation's main claims without a closer reading, the Appendix D 'alias method for negative sampling' looks like it might be a nice standalone performance improvement to Gensim (& others') negative-sampling code.

Indeed it is a significant performance boost. I believe I was the first to suggest using the alias method for w2v. I had extended it to work in a distributed setting a few years back [1] which allowed me to scale the problem to a dataset of 1 trillion words on a lexicon of 1.4BN words (roughly, the top 40 billion web pages of Yahoo's search engine).

[1] https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14956

Reply on news.ycombinator.com