Advancing NLP with Efficient Projection-Based Model Architectures | Hacker News Reader

Advancing NLP with Efficient Projection-Based Model Architectures

ai.googleblog.com

96 pointspanabee12 comments

This is the exciting moment that I´m expecting. A transition to more efficient models that are near enough to SOTA

Absolutely. ML on Edge will be one of the defining technologies of this decade. Comms use many orders of magnitude more energy than chip operations and sensor reads on embedded devices. The key is to avoid sending all that data to ML models on the cloud and instead inference on the device itself.

> that are near enough to SOTA

Only on one task of text classification by 7 categories.

New word vector scheme that learns segments of words rather than unique words... this cuts down on size of vocab and size of soft max at the end?

No expensive attention mechanism? This cuts down on number of parameters needed quite a bit, especially when sequences get very long.

I know what an RNN is but what is a QRNN?

Will need to read this more closely. I thought the key research direction now was improving the efficiency of the self attention block but maybe not.

Quasi: https://arxiv.org/abs/1611.01576

I am more excited about this in some ways than GPT-3. GPT-3 is amazing, but I can actually use a small high performing model.

I recently found this paper[1] claiming near GPT-3 performance with only a fraction of parameters. They seems to simply reformulate the input sequence to change classification to a sequence generation task.

Disclaimer, I am not affiliated to any of the authors

[1] https://arxiv.org/pdf/2009.07118.pdf

Are there any NLP techniques that are not neural net based?

Lots and lots. NLP covers a very wide range of techniques. Even some 'fancy' techniques such as sentiment analysis can be done relatively simply without neural nets (eg VADER). On the low end tokenizing is a NLP technique eg. r"\w+"

Yes, but right now there is no technique not making use of Deep Learning that is remotely competitive. It really is that clear a distinction.

Right, that's really what I was asking. Thanks.

This depends on the task. The non-neural techniques aren't making huge leaps in performance though the way neural methods are. There are tasks where traditional methods still excel or are at parity with neural methods, but the metrics haven't moved much in recent years so you don't see them in the news.

Reply on news.ycombinator.com