IMO it is a useless discussion to debate about the current SOTA or economics. In two years we will certainly have some breakthrough as we have been seen for the past years. And things are speeding up.
edit: typo.
That’s not how technical progress works. It’s not continuous. The past doesn’t predict the future. Progress may stall at any time.
I have been following closely the NLP field for over a decade and the progress is speeding up. Can it stall? Sure. But everything is pointing out that it won’t.
The difference between ChatGPT and Talk to Transformer is frankly not that large, at least when treated as a black-box. 90% of the people freaking out over ChatGPT on Twitter would have also freaked out over the original GPT, had they known it existed. The extra nuance that we're adding on feels like stalling, and while some of the optimizations have been cool (gpt-neo-2.7b, Stable Diffusion) it feels like we're hitting the top of our progress curve.
Your comment, in a nutshell and if I’m reading it right, is that it’s not worth to engage with any of these ideas because progress - whatever that means, however that’s measured - is fast.
If it were 1890 we'd be talking about how our cities will soon be buried in animal dung which will lead to the collapse of mankind. The people debating that could not have reasonably foreseen in 100 years that CO2 would be the greater risk, and 100 years from now the greater risk will be something most of us have not imagined.
Are the issues you point out worth talking about? Of course, but are they worth the amount of time and effort that we will debate them? Get back to me in 5 years and we'll see.
Gpt-3 came 3 years after transformers were invented. Now we are close to 3 years after Gpt-3, did anything nearly as big happen during that time? Things aren't speeding up at all.
In the past years we have seen a tremendous scaling of a) workforce, b) hardware, c) data. For me a breakthrough would be on data/energy efficiency? What areas are promising there?