Hacker News Reader

Comment by no_gravity | Hacker News Reader

The author argues that the number of meaningful operations a device can do per second will not keep growing exponentially. The reason he gives is that while we can add more parts that work in parallel, this has limited benefit:

    The speed up starts to disappear as silicon is left idle
    because there just aren’t enough different things to do.

I think there will be enough 'things to do'. Looking at real world applications of more processing power like AI and simulations, I expect them to be perfectly parallelizable.

View on HN

> I expect them to be perfectly parallelizable

Unfortunately, the real world does not meet your expectation. You're talking about a class of problems called 'embarrassingly parallel': https://en.wikipedia.org/wiki/Embarrassingly_parallel

Only a very tiny proportion of problems fit into that category. AI and simulations tend to be more easily parallelizable, but CPU-CPU communication is slow at any scale and imposes a limit to how much parallelism can be exploited.

The original statement is accurate:

> The speed up starts to disappear as silicon is left idle because there just aren’t enough different things to do

Efficient parallel algorithms require a problem that can be divided into smaller independent components and solved separately. Not every problem can be divided in this way. Crypto algorithms are the classic example; there's no way to perform round N+1 without first performing round N, and this is very much by design.

I wonder if those stubbornly serial problems may be arbitrarily approximated by parallel solutions.

Or a less efficient (per computation) parallel version, that scales.

A parallel algorithm is always less efficient (per unit of computational resources) due to the extra communication overhead. That's the tradeoff for being able to utilise more total computational resources.

Sure; I meant a different algorithm. e.g. inefficient by recomputing the same result in some cases; perhaps like parallel searches that may overlap. Such strategies would not have been researched much because they seem not just inefficient but stupid. But when you have cores to burn, they may turn out to be clever...

I know of a case where similar happened when silicon became much cheaper several decades ago... the inventor said it just wouldn't have occurred to anyone before then, because it required an infeasible amount of silicon. People couldn't (or wouldn't) think in those terms.

> Only a very tiny proportions fit into that category

Actually, it is exactly the other way round: most problems in the real world are about local communications of cells in a 3 dimensional lattice. Including, relativistic limits to communication. So an ideal architecture for that is a 3D cellular automaton. With bigger cells, you've got nodes on a 3d mesh (torus). Incidentally, the topology of most supercomputers.

> most problems in the real world are about local communications of cells in a 3 dimensional lattice.

Uh?

Multi-cellular life, fluid dynamics, etc. The physical world generally.

> Only a very tiny proportion of problems fit into that category

I think more correctly, only a tiny proportion of solutions to problems fit into that category, and I think that's largely because of the street light effect. The entire history of math and science has been written by people working out problems in their head, or on paper, which is more or less a serial process. People just don't think well in parallel, and until recently, neither did computers. We have a lot of serial solutions because that's where the light was, basically.

I think we're going to start finding a large variety of problems that turn out to have parallel solutions -- including a lot of problems that we didn't even know had solutions.

https://en.wikipedia.org/wiki/Amdahl%27s_law

https://en.m.wikipedia.org/wiki/Gustafson%27s_law

Sadly, in the real world most silicon for users will be busy rendering a cat trying to fit into a box and then falling over or alternatively attempting and failing to properly layout a word document while also accommodating macros inside said word document.

Word documents and tumbling cats don't really benefit from more than a few cores.

> Sadly, in the real world most silicon for users will be busy rendering a cat trying to fit into a box and then falling over

Better than ideological proselytizing and political propaganda.

On the other hand they don't need more than one good core.

Amdahl’s law

Reply on news.ycombinator.com