I'd speculate OpenAI hires a lot of people to make new data for them. There's stories of people who were interviewed where the job is just solving questions to be fed to GPT-4. They're likely looking at things that were thumbed down and adding several other data points to handle those.
There's also the quality of this data. You could feed it data from private convos (which Meta probably does) but most of that data would be low quality.
We're finding that Claude, GPT, Llama, even DeepL consistently mistranslate "you" in Indonesian as "Anda". It's never capitalized in the real world and it's a safe word, the equivalent of "they/them" pronouns in English. Imagine how weird it would be if someone wrote "Alice walks Their dog every afternoon." It's likely they were trained on government docs or perhaps some kind of ads.
But the point is there's lots of holes in the data and the perception of quality is how often it doesn't fail these.
None of the benchmarks catch this bug either. It's not just Indonesian - most of the code benchmarks use Python. So we'll also likely need better benchmarks in the future.
If we just decided to charge ahead and keep upping the parameter counts of these models by orders of magnitude, we’d probably see big improvements in the quality of the models, but is it worth it? Inferencing even at current model sizes is already notoriously expensive, and training is also both expensive, resource constrained (see OpenAI recently signing the deal with Oracle for more DCs) and technically difficult because you’re now doing training across entire data centres.
It seems reasonable for these labs to be focusing on fundamental improvements and research, while making incremental improvements to existing models at least until the compute catches up and it becomes economically feasible to just jam up parameter counts again.
Start with his short intro to LLMs, only 1 hour long but is dense with valuable stuff: https://www.youtube.com/watch?v=zjkBMFhNj_g
Also the size of the model itself isn't the only factor that determines performance, LLama 3 70B outperforms LLama 2 70B even though they have the same size.
> nothing in this paper should be interpreted as claiming that large language models cannot display emergent abilities; rather, our message is that some previously claimed emergent abilities appear to be mirages induced by researcher analyses
imo their paper's position is
- downstream task metric jumps in accuracy don't imply sudden changes in pretrain loss
- you can cherry pick metrics to show the sudden appearance of emergent abilities
- model scale is not enough to explain emergent abilities because you can scale up one model family and not get them
Their follow-up paper [0] has an appendix that touches upon how:
- it's not impossible for continuous metrics to undergo sudden increases of performance
- therefore not all sudden increases of performance are a consequence of using discontinuous metrics
- therefore emergent abilities could still actually exist
[0] Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive? https://arxiv.org/abs/2406.04391
So now a few releases after that, during which they bumped specs and made smaller changes to the architectures, there's not much room to easily grow.
More training data availability and quality is also a limiting factor to improvement (speed and resource needs can be improved more easily than that).
Also for comparison, GPT-2 was in 2019. GPT-3 in 2020. ChatGPT (3.5) was late 2022. GPT-4 in early 2023. GPT-4o was in mid 2024. I think some people drew an exponential graph from the rapid release of 3.5 to 4.
But progress stalled a little on GPT-3, which was the most groundbreaking thing in AI in decades. ChatGPT was just a dumbed down version of GPT-3, designed for the public. GPT-4 was also groundbreaking. Depending on how we connect the dots, we'd get GPT-5 from OpenAI in 2026 or later. But seeing how fast the competition is catching up, we'd expect someone else to surpass OpenAI by 2025?
I can’t predict the future, but I wouldn’t be surprised if Gemini takes the lead during 2024.