Even if LLMs didn't advance at all from this point onward, there's still loads of productive work that could be optimized / fully automated by them, at no worse output quality than the low-skilled humans we're currently throwing at that work.
This sounds a bit silly. More training will generally result in better modeling, even for a fixed amount of genuine original data. At current model sizes, it's essentially impossible to overfit to the training data so there's no reason why we should just "stop".
That in itself is a goalpost shift from
> > We will run out of additional material to train on
Where it is implied "additional material" === "all data, human + synthetic"
------
There's still some headroom left in the synthetic data playground, as cited in the paper linked:
https://proceedings.mlr.press/v235/villalobos24a.html ( https://openreview.net/pdf?id=ViZcgDQjyG )
"On the other hand, training on synthetic data has shown much promise in domains where model outputs are relatively easy to verify, such as mathematics, programming, and games (Yang et al., 2023; Liu et al., 2023; Haluptzok et al., 2023)."
With the caveat that translating this success outside of these domains is hit-or-miss:
"What is less clear is whether the usefulness of synthetic data will generalize to domains where output verification is more challenging, such as natural language."
The main bottleneck for this area of the woods will be (X := how many additional domains can be made easily verifiable). So long as (the rate of X) >> (training absorption rate), the road can be extended for a while longer.
It isn’t the panacea some make it out to be, but there is obvious utility here to sell. The real argument is shifting towards the pricing.
The world is more than just text. We can never run out of pixels if we point cameras at the real world and move them around.
I work in robotics and I don’t think people talking about this stuff appreciate that text and internet pictures is just the beginning. Robotics is poised to generate and consume TONS of data from the real world, not just the internet.
Nobody talks about it, but ultimately the strongest driver for terrascale compute will be for mathematical breakthroughs in crypography (not bruteforcing keys, but bruteforcing mathematical reasoning).
The companies might also downgrade the quality of the models to make it more viable to provide as an ad supported service which would again reduce utilisation.
And probably for the slightly more skilled email jobs that have infiltrated nearly all companies too.
Is that productive work? Well if people are getting paid, often a multiple of minimum wage, then it's productive-seeming enough.
Why are there still customer service reps? Shouldn’t they all be gone by now due to this amazing technology?
Ah, tumbleweed.