Generalized Language Models

Just came to post this link as well. Progress in machine translation has truly been amazing. Also check out the recent work from Google on Transformer-XL

https://ai.googleblog.com/2019/01/transformer-xl-unleashing-...

And for deep background, these recent sets of notes from Stanford's Deep Learning NLP class:

http://web.stanford.edu/class/cs224n/

https://www.youtube.com/playlist?list=PL3FW7Lu3i5Jsnh1rnUwq_...

Hmm, the article's summary table showed "only the top task layer(s)" for BERT fine-tuning. Is that correct?

The original paper [1] wrote:

"All of the parameters of BERT and W are fine-tuned jointly to maximize the log-probability of the correct label."

[1]: https://arxiv.org/abs/1810.04805