I guess this LoRA is the missing piece.

NVIDIA stated recently that GPT bots will become one million times more powerful in ten years. Many people doubted that.

With LoRA, I see a much higher improvement. These guys claim a 10000 times reduction in parameter size. A different way to look at it, is that with the current hardware you can train a model that has 10000 times more parameters. If you add a 100x improvement in hardware in 10 years (not at all unrealistic), that's the million. But we will have significant improvements in training methods too.