This is why parallel compute (multi core/GPUs) has taken off since 2006 - heat puts a cap on clock speed, so the only option is to do more operations per cycle. Even some microcontrollers are dual-core now.
I don't see this trend stopping. Massive parallelism is the future of computing.
Also, even if Moore's law remains in place for now, it will end at some point and the world needs to think about what happens afterward.
From my dealings with parallel algorithms, there is much less sequential dependency than I think most people expect.
The brain is massively parallel - 80 billion neurons operating asynchronously at only 100hz.
>is having a neural network the end problem that needs solving?
Absolutely! Neural networks are a general-purpose way to solve a huge number of problems, especially problems that involve raw data or the messy real world.
You contradicted yourself in that last paragraph. I don't know what else to say to you about that.
Parallelism allows you to do computations that integrate huge amounts of information, like checking every pixel of an image against thousands of weak rules all at once. This is very inefficient to do serially because you can only check one at a time. It's also a huge usecase right now because machine learning lets you learn weak rules from data.
Of course there may be some practical limit to Gustafson's Law, but I don't think we've found it yet, at least in many scientific domains.
Price increases 50% every 2 years