Hacker News Reader

Comment by nuc1e0n | Hacker News Reader

There are limits to how much programs can be parallelized, see Amdahl's law. Even if Moore's law technically continues, the extra capacity isn't adding the same amount of extra value to the workflow of end users that it once did. Here's a video that talks about diminishing returns as it relates to graphics in computer games https://m.youtube.com/watch?v=VUCYAws6oW0. Diminishing returns in regards to polygon counts themselves have also been discussed: https://www.polygon.com/2013/12/10/5192058/opinion-stop-dwel...

Also, even if Moore's law remains in place for now, it will end at some point and the world needs to think about what happens afterward.

View on HN

Note that Amdahl's law doesn't apply if you change the entire algorithm (program) to fit better on parallel hardware (i.e. the task is not fixed).

From my dealings with parallel algorithms, there is much less sequential dependency than I think most people expect.

What day to day problems do people have that require parallel processing to solve anyway? Sure graphics and neural networks, but is having a neural network the end problem that needs solving? Maybe I'm just a grumpy old programmer but I believe the human mind is specialised to think in terms of serial solutions. I think that's no accident as race conditions and other synchronization issues are difficult to avoid in parallel algorithms. That may be a fundamental property of processing in general, even for algorithms some AI may come up with, which remains mostly fictional at this point.

Compilation comes to mind. Efficient distributed compute also usually prefers vertical scaling (i.e. more cores per node).

>I believe the human mind is specialised to think in terms of serial solutions

The brain is massively parallel - 80 billion neurons operating asynchronously at only 100hz.

>is having a neural network the end problem that needs solving?

Absolutely! Neural networks are a general-purpose way to solve a huge number of problems, especially problems that involve raw data or the messy real world.

I'm talking about the solutions we tend to produce, not our internal workings.

You contradicted yourself in that last paragraph. I don't know what else to say to you about that.

I know I think in parallel often enough that two threads end up disagreeing with each other and a 3rd one has to sort it out. I can’t begin to describe the internal fork() sensation that happens in my noggin constantly.

This is true, but parallel programs are not more limited - they're just good at different things. We will write different kinds of programs to do different kinds of tasks because of access to massive parallel compute.

Parallelism allows you to do computations that integrate huge amounts of information, like checking every pixel of an image against thousands of weak rules all at once. This is very inefficient to do serially because you can only check one at a time. It's also a huge usecase right now because machine learning lets you learn weak rules from data.

It’s funny, because while technically correct, you may end up fundamentally wrong that there was a limit to the value we might find in parallel compute. After all, we may have lie of sight to AGI through parallel scale alone.

What you've identified is known as Gustafson's Law. If you can keep solving larger and larger problems, your solution time doesn't necessarily go down, but "speedup" is potentially uncapped.

Of course there may be some practical limit to Gustafson's Law, but I don't think we've found it yet, at least in many scientific domains.

Reply on news.ycombinator.com