Hacker News Reader

Good overview.

At the other extreme, some recent works [1,2] show why it’s sometime better to scale down instead of up, especially for some humanlike capabilities like generalization:

[1] https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00489...

[2] https://arxiv.org/abs/1906.04358

View on HN

If greater parameterization leads to memorization rather than generalisation it's likely a failure in our current architectures and loss formulations rather than an inherent benefit of "fewer parameters" improving generalizaiton. Other animals do not generalize better than humans despite having fewer neurons (or their generalizaitons betray a misunderstanding of the number and depth of subcategories there are for things, like when a dog barks at everything that passes by the window).

Reply on news.ycombinator.com