Comment by cs702 | Hacker News Reader

Comment by cs702 | Hacker News Reader

This kind of "blocking-and-tackling" work is important.

The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks.

Remarkably, the performance of a Transformer in the tested tasks improves log-linearly with the number of parameters, suggesting that even the largest model tested, with 1.5B parameters, still underfits 40GB of text.

This is compelling evidence that we do NOT need new architectures, NOR new kinds of training objectives, NOR new theories, for better language modeling! We can get better language modeling simply by increasing model capacity (i.e., by adding more parameters to existing models), which becomes easier and simpler to do as hardware continues to improve over time.

Great work.

PS. In case it's not clear: I'm not saying we should suddenly stop searching for new, better ideas and architectures. That would be silly. Please don't attack a straw-man :-)

[a] https://arxiv.org/abs/1706.03762

> …we do NOT need new architectures…

You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that works remarkably well – akin to how Mikolov et al. reconsidered how to learn vector representations back in 2013 [3] to ingest magnitudes more data than previously possible.

[1]: http://www.aclweb.org/anthology/D/D16/D16-1244.pdf

[2]: http://papers.nips.cc/paper/7181-attention-is-all-you-need

[3]: https://arxiv.org/abs/1301.3781

I would not call myself a model-focused researcher so I would love to play down their importance, but to be intellectually honest one should call out hyperbole where ever one sees it.

The point I was trying to make is that we can get better language models with more computation, not that we should stop researching new ideas and architectures. In hindsight, perhaps the language in my post wasn't sufficiently clear on this. See my PS. You and I are not in disagreement :-)

I do not get it. If what you are trying to say is: “more data/computation helps”, then why even bother throwing in models, training objectives, and theory into your statement? We have known that more data helps for at least two decades, so saying this is stating the obvious. Are you then trying to say: “Right at this very moment we can push a bit further with more data/computation given what we have for this specific task”, well, then again this is nothing particularly insightful to bring to the table as surely this occurs at least once a month for a given task/model category.

This is not a matter of clarity, it is a matter of form – drop the hyperbole and speak specifics rather than trying to appease both sides of a fictitious divide.

Agreed, the transformer's approach to memory/attention is certainly novel and a non-trivial departure from RNNs. It's also questionable the extent to which this is not over-fitting. Some are already finding phrases on the web the model is clearly stitching together. This is not to underplay the impressiveness of stitching the correct phrases and n-grams in a thematically coherent way for paragraphs at a time, it's still very impressive (this transformer should be nicknamed Bumblebee).

Rather than under-fitting, I'd consider this diminishing returns in the model's ability to make use of the available information. You can get within 10% of it on the challenging winograd schema task using only 2%-5% of the used data. Many here are programmers and computer scientists, rather than stoop to hyperbole we should try to understand. What class of automata can this model learn? Finite state machines, push down automata? Transformer has memory but no loops so I very strongly doubt them to be more than that and yet they perform so well on language (but then again, not simple program learning) tasks.

Do u have any recent links to program synthesis (with transformer or other novel attempts ?)

Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity. Humans don't need to read through 40 GB of text multiple times to learn to write.

I guess the data requirements act as a stand-in for our genetic evolution... Our brain models are good at learning the things we learn. Our computer models aren't yet.

/tautology

> Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity.

Yes, agreed. Nothing I said above contradicts that! :-)

> Humans don't need to read through 40 GB of text multiple times to learn to write.

Yes, that's true... but to keep the comparison fair, note that we do need many years of schooling to learn to read, say, at a high-school or college level. And before learning to read, we first must learn to speak, which surely helps. And we also get to inhabit bodies that see, smell, touch, and interact with the physical objects that we read and speak about during our formative years, which also helps. The more one thinks about it, 40GB of data is actually a tiny figure in comparison to the amount of training data that flows continuously to our brain from all senses. I think I read once that our brains process on the order of 10 to 100 GB of training data per second.

Conscious processes are estimated to work on the order of 10^2 bits. Vision, at the retina, is estimated at 10^7 bits/sec. It drops another order of magnitude by V1. Also note that long as they're not isolated, a deaf and blind person has no trouble getting to full human reasoning ability despite being vastly more impoverished in available data compared to the average person.

A human will also be learning vision, hearing, walking, physics, causal reasoning, and much more. This comparison just isn't well grounded. Task specific is how much training does a young brain require to learn to produce language? If the brain comes with innate advantages then rather than be resort to inefficiency and excusing our models, we should try to see if they can be bettered.

I'm not well versed at all in signal theory, so I'm genuinely curious how these bitrate estimates are made, and would love to see the source of these specific numbers.

How do you estimate the effective (digital) bitrate of an inherently analogue system?

After 40 GB of text the model doesn't know anything about how the world works, and it shows many times in the examples. Nobody would do some of those mistakes, not even young kids. Other mistakes are more subtle but still show a total lack of understanding.

Then yes, it's enough to write text that nobody really cares about and that could cover a lot of what we read.

It's because nobody dumps 40GB on kid. Kids go through long process of feedback-corrections. I imagine if there would be some crowd funded project to provide feedback about mistakes to this model, it would learn and produce better results fast.

40 GB is surprisingly close. I estimate I already read at least 4 GB of text so far. That's just 10 times more samples. I probably write better than GPT-2, but certainly not faster.

4GB of English? 500 words per single-spaced page, 5 letters on average per English word, so 2500 bytes/page.

4 gigs then would be 1,600,000 pages.

That's 219 single-spaced pages per day every single day for 20 years straight. High, but I guess not outside the realm of possibility depending on the complexity of the text.

For perspective, 219 pages amounts to about a railway novel in a day, or about two ~800-page fantasy doorstoppers in a week. At a rule of thumb of about a page per minute for light reading, it's just under four hours: a hefty investment, but well within the realm of possibility. High compared to the general population, but table stakes for a book club or otherwise avid reader.

I'd've easily doubled or tripled that over the summer as a teenager.

I think your conclusion is too strong. Yes, we know bigger models and data generally lead to better performance (e.g. BigGAN results last year) but progress in architecture can still speed up progress in ML tasks. If we were still stuck using RNNs and LSTMs for language modeling we wouldn't be talking about this news today.

I feel like my human brain underfits a lot of data.

My human brain definite zero-shot transfer learns almost everything.

Reply on news.ycombinator.com