According to Ilya Sutskever: "results from scaling up pre-training have plateaued". https://www.reuters.com/technology/artificial-intelligence/o...

They're trying other techniques to improve what we already have atm, but we're almost at the limit of its capabilities.