It's kind of like saying Redis is just an dict. The optimizations in the entire software/hardware stack is what makes these innovations possible. A C implementation would not work in reality for large models.
I don't disagree that hardware acceleration is key in enabling these models, but I still find it interesting how simple the core techniques are.
You mean CPU implementation. C language isn't the culprit here, and low-level parts of ML frameworks are normally written in C or C++ as well and compiled with nvcc or equivalent.