> This paper addresses the challenge by asking: how can we trade off more compute for less data? 

Autoregressive models are not matched by compute and this is the major drawback.

There is evidence that training RNN models that compute several steps with same input and coefficients (but different state) lead to better performance. It was shown in a followup to [1] that performed ablation study.

[1] https://arxiv.org/abs/1611.06188

They fixed number of time steps instead of varying it, and got better results.

Unfortunately, I forgot the title of that ablation paper.