More generally though, the fact that matching GPT-4 is the marketing blurb as opposed to a GPT-3 -> GPT-4 level transition has me wondering: are we seeing the limits of current "just scale the data" approaches to ML? Like, we've basically scraped everything worth scraping off the web, and when you do that with any of our current techniques, it lands you somewhere around GPT-4?
I don't have the link anymore, but i saw a quote from a LinkedIn connection recently at one of the FAANG's about how data matters so much more than architecture for ML today. To paraphrase, it went something like "it turns out that when your data is large enough, architecture doesn't matter. GANs converge to about the same thing as diffusion models. Convnets converge to about the same thing as ViTs. Data size and quality matter a lot more".