As someone who has been researching neural networks in a variety of settings for a very long time now, it is actually pretty obvious. There is also no real "magic" to it, even though it certainly might seem so to people who did not follow the academic world of research closely. But to those who do, all of this followed a pretty straightforward path, even though certain key steps were only obvious in hindsight. We already knew since the 90s that a perceptron with a single hidden layer can approximate any function (with some caveats that in practice only boil down to computational limits) with arbitrary accuracy, with the error scaling like 1/N with the number of hidden Neurons. But the proof of that theorem already shows that this is by far not the most efficient way to approximate functions. While in practice you could plug pixel values of an image directly into a perceptron, computationally it turned out to be hugely more efficient to use convolutions first as a dimensionality reduction scheme. This not just allowed people to train much larger networks on larger datasets, it also highlighted how additional layers enable hierarchical knowledge. So the first layer of such a network might only encode lines or circles, while deeper layers could encode noses and ears and eventually entire human faces. For language modeling, the thing holding everything back was also computability. Recurrent neural networks are theoretically even more powerful than simple perceptrons, but they come with a significant cost when computing gradients. Trying to improve these restraints is what eventually led to the transformer, which at its core is just an extremely scalable, general purpose, differentiable algorithm that you can optimise using backpropagation. We didn't need this architecture from a purely theoretical perspective, but we needed it in practice because our computing hardware is still very limited once we are trying to mimic actual biological neural networks as you would find them in the human brain.