They didn't solve any new problems. They took old ideas and just cranked the dial all the way up to 11. Then an entire market formed around building the hardware necessary to crank it even further. This all represents a revolution in computing power not in what neural net folks were doing at all.
> Faith and Vision are truly spiritual things, even in tech.
Yea, but money isn't.
The reality is that we had to solve vanishing / exploding gradient problem. That wasn't accomplished by getting faster compute, but by devising better architectures and using activation functions with properties we needed.
That's how ANNs got out of the "somewhat useful" into "really useful" categories. Hardware helped move it even further. But without the work of computer scientists and mathematicians we would never get to the point when the hardware matters.
I was working on an alternative approach of using much smoother activation functions at much higher precision, 64 bits in production using GPUs and 128 in testing using CPUs. It worked well enough that I don't think the field would have slowed down much. The only issue is that we'd be working on networks that are 10 times smaller.