Hacker News Reader

Comment by tehsauce | Hacker News Reader

I think it's important to notice that if we're using the metric of "300,000x" increase in computing power applied to ML models, the giant increase has mostly been due to parallel computing playing catchup on decades of moores law all at once. It will hit a wall and die with moores law fairly soon. Physics requires it.

View on HN

How is parallelism limited by physics?

I thought the point of parallelism is you can throw more chips at a problem and see improved performance. Single chips are limited by physics, but true parallelism scales linearly ad infinitum.

Can anyone with more knowledge than me speak to known limits of parallelism? I’d guess it’s not truly infinitely scalable.

You can't scale linearly ad infinitum because eventually the communication (i.e. memory) cost gets too high.

This reminds me of a thought experiment I heard from -- if memory serves -- Scott Aaronson. The gist is that the fastest super-computer will be on the edge of a black hole. If you run any faster, there will be too much energy concentrated on a given area, thus creating a black hole. Similarly, when you run so many parallel devices (on GPU, CPU, etc) together, you will want to put the devices as close to each other as possible (speed of light limits the rate of communication). You then pump too much heat into a small area, and getting so much heat out is, among other things, a physics problem.

That's a very far limit, though. It will not have practical consequences for a long time.

Also, if you don't squeeze as much as you can into a small space, you can scale sublinearly ad infinitum (in practical terms, which don't include heat death of the universe).

Parent is probably referring to amdahls law - which limits speedup in parallel computing systems https://en.m.wikipedia.org/wiki/Amdahl%27s_law

That doesn’t really apply in this case though because the major thing people are using the increase in parallelism for is running larger computations or more parallel computations of the same size, rather than trying to run the same computation in less time.

If you built a computer with a squillion chips that was a light-year long, it would take a year at minimum to get a message from one side of the computer to the other. The same issue applies on a smaller scale for smaller computers

Reply on news.ycombinator.com