There's a ton of recent work on data curation / synthetic data generation that shows that smaller high quality datasets go a lot further than scaling up on noisy web data.

The scaling law plots are log scale so to get more juice with naive scaling we'd need to invest exponentially more resources, and we're at a point where the juice is not worth the squeeze, so people will shift to moving the curve down with new architectures, better curated datasets and test time compute / RL.

See:

- FineWeb: https://arxiv.org/abs/2406.17557

- Phi-4: https://arxiv.org/abs/2412.08905

- DataComp: https://arxiv.org/abs/2406.11794