According to Ilya Sutskever: "results from scaling up pre-training have plateaued".

https://www.reuters.com/technology/artificial-intelligence/o...

They're trying other techniques to improve what we already have atm.