Hacker News Reader

Comment by patelajay285 | Hacker News Reader

This is interesting work. We at Plasticity (YC S17) have open sourced something similar called Magnitude (https://github.com/plasticityai/magnitude) a few months ago for querying vector embeddings quickly with low memory usage using SQLite and a standard universal file format (.magnitude) between word2vec, fastText, and GloVe. We also added features for out-of-vocabulary lookups, bulk queries, concatenating embeddings from multiple models, etc. It may also be of interest to folks looking at this.

We also have a paper on Magnitude we will be presenting at EMNLP 2018 and ELMo support is coming soon!

View on HN

SQLite (and SQL in general) is ridiculously slow for specialized applications like this. http://www.lmdb.tech/bench/microbench/

It depends what you are optimizing for. It's a little slower for random disk reads, but it doesn't matter given we utilize an in-memory LRU cache and most text has a Zipf-ian distribution which will hit the cache very often even with a small cache size. (see my comment elsewhere on this thread)

Developers are more familiar with SQLite, it's bundled with Python, it is easy to add to nearly every language, and it has specialized features we use like the Full Text Search module for out of vocabulary lookups.

Wrappers for LMDB are available for nearly every language as well. https://symas.com/lmdb/technical/#wrappers

Oh wow, how did I not know about this :) Looks super useful.

How would ELMo work if a neural network needs to be run?

Thanks! Still working on it, but it will embed the ELMo model in the .magnitude file, wrap the ELMo model interface code so that it is standardized with Magnitude's features (querying, concatenating, iterating, etc.) and will use ELMo's OOV method natively.

Note also:

* reddit (Storage and retrieval of Word Embeddings in various databases): https://www.reddit.com/r/LanguageTechnology/comments/9cf0zk/...

* ... links to this GitHub repo: https://github.com/martialblog/word_embedding_storage

I have tried plasticity (even submitted a PR*) before putting together this repo but found it horrendously slow in comparison. It really does depend though; as plasticity utilise an in-mem cache if you're querying on many similar words repeatedly, that is likely to be faster.

EDIT: sorry it wasn't a PR but an issue

Reply on news.ycombinator.com