Deep Learning, NLP, and Representations

Everything on this guy's blog is fantastic. I love the post on neural network manifolds (http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/).

When I experimented with word vector representations (word2vec), I found that the analogous embedding part was a bit oversold in a subtle way. While some sorts of analogies worked (the easiest to see was gender: woman-man+boy=girl), other sorts of analogies that appeared just as straightforward to a human were not at all understood by the embeddings.

For example,

water-boat+plane=potable_water (air, sky, etc nowhere to be found).

small-big+tall=taller (short, etc nowhere to be found).

shell-pecan+coconut=shells (husk nowhere to be found).

What I eventually realized is that the vector representations only really work in the case where an averaging operation would get the right answer anyways. For example, woman-man+boy= girl, but you can also get that through (woman+boy)/2=girl. (or even woman+boy=girl) In my experimentation I have yet to see a case where the subtraction aspect is actually necessary to recover the relationship. This would indicate that the vectors are only capturing similarity between words, and not and not any second order relationships between them as is commonly claimed.

Honestly, I wish I could discuss this directly with someone doing research in this area, because it seems to me that the vector representations really aren't capturing relationships in the way they're touted. I think it's an interesting possibility, and I haven't seen it addressed in any of the literature I've read.

EDIT: On further investigation, it looks like the google news vectors maybe just aren't that good? I wasn't able reproduce hardly any of the analogy examples given in the table "Relationship pairs in a word embedding." from the OP.

Always excellent content from Christopher Olah. So few people do such a clear job of explaining these concepts -- this is technical writing done right.