But I think you are selling GPT a bit short. Partly because transformers are incredibly general architectures, which make few assumptions about the domain. They are basically SOTA on every domain for which learning-based methods are applicable.
And secondly because ChatGPT can perform “in-context learning”. That is to say, purely by scaling next-token prediction, there is an emergent phenomena where GPT gains the ability to learn from the prompt it is given, and solve new tasks outside of the training distribution.
Hattie Zhou has a cool paper showing how you can teach GPT addition by prompting - a toy example, but it signifies something profound in my opinion. In the last 6-months I have shifted my own view from “scaling is a waste of compute” to a recognition that LLMs are a serious leap forward.
For something very hyped, I think it’s still under hyped and under appreciated.