Microsoft and Nvidia have created a 530B parameter language model
microsoft.com
microsoft.com
The blog post doesn't provide a lot of numbers but looks like it beats the state-of-the-art in a couple of commonsense reasoning benchmarks.
Still, it shows that you can't just keep scaling the models and expect magic.
They undertrained it, and the real benchmarks will be in the paper, so it's not at all obvious that this is not exactly what you would predict from existing scaling laws on GPT models (remember, they are power laws / log curves, so 3x doesn't move the needle that much even if it had been trained optimally). Plus of course, the history of these sorts of models like GPT-3 or CLIP is that the benchmarks are uninformative and you only get an idea of what improvements have emerged once the users get their hands on it and are able to discover the new induced capabilities. Nvidia/MS sometimes release their models, but mostly don't, so who knows...
Anyway, this is a redundant submission to https://news.ycombinator.com/item?id=28827957