Thanks for your blog post!
Are you familiar with http://www.offconvex.org/2016/07/10/embeddingspolysemy/?
I read a blog post maybe a year ago that explained word embeddings under an 'atom interpretation', in which word embeddings are really sparse combinations of 'atoms' (I don't really remember more than that). It was very interesting, but then I forgot about it. Trying to find the post again I only came up with the above paper, which is probably the same idea, but it wasn't what I originally read. Wish I could find it.