Hacker News Reader

Comment by mikewarot | Hacker News Reader

My understanding was that Word2Vec[1] was trained on Wikipedia and other such texts, not artificially constructed things like the triplets you suggest. There's an inherent structure present in human languages that enable the "magic" of embeddings to work, as far as I can tell.

[1] https://code.google.com/archive/p/word2vec/

View on HN

Has there been any rigorous evaluation of word2vec calculating 'king-man+woman=queen' associations? I only recall the author providing some cherry-picked examples from their results, which I suppose makes it a seminal AI paper.

The original paper included source, and that has their test data and results -- it gets ~77% accuracy on about 20k example word analogies (with 99.7% coverage), and 78% accuracy with phrases with 77% coverage (and a much smaller data set, 3,218 examples). You can see the test sets here:

https://github.com/tmikolov/word2vec/blob/master/questions-w...

https://github.com/tmikolov/word2vec/blob/master/questions-p...

and you could see how much better LLMs do on the same 20k examples.

A lot more info about how it worked here https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935... and an interactive demo https://dash.gallery/dash-word-arithmetic/ (blog post about the demo https://medium.com/plotly/understanding-word-embedding-arith...)

If a paper attacks plus and minus it ought to cite this one.

Ah, yes I may have mis-remembered or misunderstood when looking at the training data and model definition, it would have been about a decade ago now. Or perhaps I was thinking of an unrelated experiment to use chosen analogies during training but yeah that isn't part of the original paper.

The training process from the original Mikolov et al. paper only uses the analogy examples (questions-words.txt and questions-phrases.txt) to measure accuracy after training: https://github.com/tmikolov/word2vec/blob/20c129af10659f7c50...

There is an inherent structure in language. Embeddings do not and will not capture it. It's why they do not work. Their ability to form grammatical sentences with high accuracy is part of the illusion that you have been understood.

Reply on news.ycombinator.com