Hacker News Reader

Comment by ninjin | Hacker News Reader

> …we do NOT need new architectures…

You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that works remarkably well – akin to how Mikolov et al. reconsidered how to learn vector representations back in 2013 [3] to ingest magnitudes more data than previously possible.

[1]: http://www.aclweb.org/anthology/D/D16/D16-1244.pdf

[2]: http://papers.nips.cc/paper/7181-attention-is-all-you-need

[3]: https://arxiv.org/abs/1301.3781

I would not call myself a model-focused researcher so I would love to play down their importance, but to be intellectually honest one should call out hyperbole where ever one sees it.

View on HN

Agreed, the transformer's approach to memory/attention is certainly novel and a non-trivial departure from RNNs. It's also questionable the extent to which this is not over-fitting. Some are already finding phrases on the web the model is clearly stitching together. This is not to underplay the impressiveness of stitching the correct phrases and n-grams in a thematically coherent way for paragraphs at a time, it's still very impressive (this transformer should be nicknamed Bumblebee).

Rather than under-fitting, I'd consider this diminishing returns in the model's ability to make use of the available information. You can get within 10% of it on the challenging winograd schema task using only 2%-5% of the used data. Many here are programmers and computer scientists, rather than stoop to hyperbole we should try to understand. What class of automata can this model learn? Finite state machines, push down automata? Transformer has memory but no loops so I very strongly doubt them to be more than that and yet they perform so well on language (but then again, not simple program learning) tasks.

Do u have any recent links to program synthesis (with transformer or other novel attempts ?)

The point I was trying to make is that we can get better language models with more computation, not that we should stop researching new ideas and architectures. In hindsight, perhaps the language in my post wasn't sufficiently clear on this. See my PS. You and I are not in disagreement :-)

I do not get it. If what you are trying to say is: “more data/computation helps”, then why even bother throwing in models, training objectives, and theory into your statement? We have known that more data helps for at least two decades, so saying this is stating the obvious. Are you then trying to say: “Right at this very moment we can push a bit further with more data/computation given what we have for this specific task”, well, then again this is nothing particularly insightful to bring to the table as surely this occurs at least once a month for a given task/model category.

This is not a matter of clarity, it is a matter of form – drop the hyperbole and speak specifics rather than trying to appease both sides of a fictitious divide.

Reply on news.ycombinator.com