> "data, not size, is the currently active constraint on language modeling performance"

This is a bit incomplete. It goes both ways. Right in the abstract, it says:

"the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled"

You are probably referring to the statement:

"We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant."

So GPT3 and co would perform better with more data, but only up to a certain point. At that point, you would also need to scale up the model size again to get better performance.

But also, GPT3 and co would perform better when they would be scaled down a bit, when you keep the same training data. That is actually what the Chinchilla paper does. Their Chinchilla model is smaller than GPT3 and others, trained with same amount of data (as Gopher), and performs better.

I don't really see a problem in scaling up models even more. This paper just says that you should also scale up the training tokens equally.

As far as I remember, this paper does not quite address whether you can use the same training data twice. I think all statements assume that every training token is only used once.