Hacker News Reader

Comment by arnocaj | Hacker News Reader

I would not consider word embeddings to be state of the art anymore. Word Embeddings are like TF-IDF when word embeddings came out. Have a look at BERT model that just recently got published and is outperforming all kind if NLP tasks with one main Architecture. I would consider BERT language model two levels higher than word embeddings, as it considers full context sensitive embeddings, dependent on the text left and right of the word in parallel.

View on HN

I would second this; sentence embeddings outperform word embeddings on basically all tasks where you actually have sentences to work with. The only downside is that they're significantly more computationally intensive, especially for Transformer models like BERT.

(Note: I'm fairly biased, since I work on https://www.basilica.ai, which among other things makes sentence embeddings available over a REST interface.)

Is BERT computationally (and sample-wise) equivalent to previous SOTA?

(I do DRL but not NLP)

I sometimes read these DL papers and the requirements are not really feasible if you have to re-implement them in a modified domain.

BERT is more computationally expensive. It might end up giving better results on the task mentioned in the paper but we don't know. At the time of writing this all of the contextual word embedding techniques were fairly new and were not tried.

I can see utility in demonstrating breakthroughs with the simpler compatible technique as opposed to a more complicated state of the art technique. The goal of scientific communication is to communicate with the simplest examples possible the rationale and effect of a proposal. Then anyone using more advanced frameworks can understand and consider implementing it in theirs.

Reply on news.ycombinator.com