There is nothing here but a promise. Back in the day we called this "vaporware".
Here's my guess: Some team under web search trained a large Transformer based model but with some adjustment here but now on a massive dataset from the crawled web pages using tons of TPUs. It made an incremental improvement to the search quality metrics and was shipped to production.
Their models accomplish ridiculously powerful things. Tbh I think it's far _more_ likely the answer is "this is crazy powerful, but the engineers didn't feel like writing a blog post about it, and the marketing team hasn't figured out how to monetize it yet".
The lack of details makes me think they're either hiding a new technique they'd rather keep secret because it provides a competitive advantage, or that it's really only a marginal improvement over existing NLP models (or an ensemble of them with nearly no improvement on any given metric) and the 1000x improvement is on a metric that no actual ML scientist would respect.
I don't have the slightest bit of information about Google's AI team to know if those are the only two options and if so which is more likely.
We have seen in the last few months attacks on Google Translate, GPT-3 and other language models from the PC crowd, including the famous AI Ethics firings. It's just tricky to show it in this climate.
The PC crowd don't believe language is fair and concepts neutral, instead saying they are an expression of systems of power. So language models are a natural target for them because they could amplify biases against their identity groups.
I find this critique hasty especially because big language models are nascent technology. We shouldn't throw away the baby with the bath water!
Figuring out how to feed language models with diverse sources of information is a tough challenge, but not impossible. I share Gebru's concern about "stochastic parrots".
And calling language model "parrots" is flouting. Many people worked for decades to reach that accomplishment, here come the critics to shit all over it.
> But how do you update a culture's shared set of beliefs?
It's not the place of AI models to do activism, and it's a slippery slope leading to AI based inquisition. Take a look at how China uses AI to oppress their own people.
I don't understand why you'd find the use of the term 'parrot' offensive. Language models extract linguistic patterns. GPTs generate patterns based on those which they have been trained on. That's a process that can be described as parroting. If you find it offensive because you think it implies that the researchers coming up with these models aren't worthy of credit, I think you are reading something into it that isn't there. At least not from my perspective.
When I mentioned updating cultural beliefs, I was referring to the traditional way of going about it: through cultural products. My point was that the "PC crowd" would be better off if they relied on this strategy rather than attempting to halt the development of language models. I was absolutely not suggesting that language models should be used to "train" members of society. That's a dystopian nightmare.
True, as long as you empathize with the story, and empathy comes from feeling united. But the PC army abandoned empathy in favor of identity, and think they are on their own, fighting a war, a zero sum game. Noninclusive politics is asking for empathy, how ridiculous, it's the same as demanding tolerance for intolerance.
They teach a whole ideology of guilt in order to dehumanize their opponents and cut the empathy towards them. They can't complain.