You could argue that BERT was a first go at it, but until transfer learning doesn't equate to "throw compute at it because we're Google/OpenAI", we're nowhere near to having solved this.
throw compute at it because we're Google/OpenAI Sorry but for training time, neural network deep learning is far from a "smart" paradigm. It is essentially statistical brute force + a few clever math tricks. This is a part of the answer on how to create an artificial general intelligence. But where's is the research for creating a causal reasoning system understunding natural language? It mostly died in the AI winter, and except a few hipsters like me or opencog or cyc, is dead. I wonder how many decades will be needed for firms like Google to realize such an obvious thing (that real intelligence is statistical AND causal).
Thus, we've seen real value out of transfer learning that doesn't require overly much compute power (and actually could even be run on free colab instances, I think).
That said, I agree that the problem is still very far from being "solved". In particular, I have a fear that most recent advances might be tracked back to gigantic models memorizing things (instead of doing something that could at least vagely be seen as sort of understanding text) to slightly improve GLUE scores.
Still, I am highly optimistic about transfer learning for NLP in general
most people also actually distill BERT to reduce the computation cost