Hacker News Reader

Embeddings can be compressed even further than that : https://arxiv.org/abs/1511.06397 (disclaimer: I'm the author)

Wow - 3 bits per element. Wonderful work with the quantization, I'd love to implement this. How would you recommend I proceed?

The paper (with fewer typos) was actually accepted into 'ICONIP' in Japan in October - so I'll definitely have code on GitHub by the end of the summer. Currently my Theano implementation is buried in typical exploratory kind of code, which just needs to be stripped away to make something functional from GloVE->Sparse in one command.

The NNSE paper has associated code already, but I found setting the sparseness preference parameter was very hit-and-miss, which is why I preferred the explicit sparse-by-percentage measure in my work.

~6% sparsity + the way that you think about data representation is very interesting. I will have to have a shot at running the autoencoder over the Mikolov 3m word corpus. My goal is to get the first 1m words compressed at under 100 MB zipped, including all indexes, which currently allows us to distribute the vectors for free at github without paying for data transfer. (To date about 8000 words have been mapped through the words2map Google API, while I haven't really begun to do anything interesting here yet...)

Reply on news.ycombinator.com