Linus: Parallel computing is a huge waste of everybody's time

He explicitly says "Parallel code makes sense in the few cases I mentioned, where we already largely have it covered, because in the server space, people have been parallel for a long time" — so most of that rant is about desktop and mobile computers, often called "client-side". Where indeed four cores seem to be more than enough for most needs.

On the "server" side, though (I hate the outdated name), the story is quite different. I can easily make use of many cores, especially with tools like Clojure, that make writing correct concurrent programs much easier.

> Parallel stupid small cores without caches are horrible unless you have a very specific load that is hugely regular (ie graphics).

And

> The only place where parallelism matters is in graphics or on the server side, where we already largely have it. Pushing it anywhere else is just pointless.

Edit: "editorialized" meaning "choosing the most extreme quote of a rant"

It's an interesting argument but rests upon all new algorithms (he brings up machine vision as an example) having dedicated hardware. Ultimately, sure, but there's still a hell of a gap between viable algorithm and dedicated mobile-ready hardware. If the pace of invention slows I'd agree with Linus.

I think the pace of invention will continue to accelerate and parallel processing on the client will be a valuable resource to have.

On other side parallel programming does not have to be hard. Fork-Join framework in Java and parallel collections in Java 8, are trivial to use and scale vertically pretty well.

And finally I think there is not actual demand for parallel programming. 99% computers have 4 cpus or less. GPUs are useless for most tasks. I have prototype of my program which scaled well to 20 cores, but nobody is interested.

> So give up on parallelism already. It's not going to happen.

> End users are fine with roughly on the order of four cores,

> and you can't fit any more anyway without using too much

> energy to be practical in that space.

End users was fine with a single core Pentium 4 on their workstation. We progressed. How would even Linus know that we won't find a way to make parallel work en masse?

We have used parallel computing quite extensively for: 1-3D, 2D vector graphics. 2-Audio processing 3-Image processing 4-All of the above(video).

We could parallelize something to be more than 100 times more efficient(ops per Watt) than on the CPU(). But proper parallelization comes at a cost: Efficient memory management is hell.

I mean, people are afraid of c manual memory management, that is nothing compared with the complexity of parallel memory management. You need semaphores or mutexes to access common memory, but the most important thing is that you need to make independent as much memory as you can from each other, replace sequential steps, etc...

So if the reason for making the kernel parallel is a 10% THEORETICAL increase, forget about it, 10% is nothing with the complexity you have to add.

() Power consumption normally increases with the square of frequency, so using more cores instead of higher frequency you get very efficient. The brain itself runs very slow but with a high amount of cores(neuron clusters).

The PC architecture started out being hampered by the CPU + dumb peripherals architecture, but this was a big departure from the norm during that era. In the "home" market, most machines either had CPUs too weak to drive the peripherals (my first "parallel stupid small cores" system was a Commodore 64 + a 1541 disk drive - you could, and people did, and wrote book about, download your own code to the 1541 and do computations on it) or explicitly used CPUs or co-processors all over the place to offload things, like the Amiga.

My A2000 had of course the 68000 (and later a 68020 accelerator), but also had a 6502 compatible core running the keyboard, a Z80 on the SCSI controller card, on top of the Amiga's custom chips which had the copper, blitter and sound chips that all had limited programmability. The irony is that it perhaps made a lot of us overly gung ho on the 680x0 line (though it does have a beautiful instruction set) because our machines felt so much faster than comparably clocked PCs - hence many of use were ok with not upgrading CPU models and clock rates as fast as in the PC world.

It was first when PC's started sprouting co-processors (graphics cards and sound cards with advanced capabilities first), that the Amiga truly lost its edge (at the same time Motorola failed to deliver fast enough versions of their newest 680x0 models, faster CPU's would have been insufficient) - until then the co-processors and philosophy offloading everything possible had compensated for the by then anaemic average CPU speeds.

Though the 3rd party expansions gave one more fascinating multi-processing step: Systems that would let you run 680x0 and PPC code on the same machine (PPC cpu on the expansion card, 680x0 on the motherboard).

PCs have been steadily sprouting more small cores. They're just not as visible.

On the server side it is extreme: I have single CPU servers at work where the main CPU may have 6-8 x86 cores, but where there may be 30+ ARM or MIPS cores when you tally up the harddrives (dual or tri-core in many cases), RAID controllers, IPMI cards, some networking hardware etc.. But these cores are getting so cheap and so small that we should expect to continue to see offloading of more functionality.

On the Amiga, this was what was our day to day reality. SCSI was always favoured, for example, because it was lighter on the CPU than IDE was (hence universal disdain for the guy that forced engineering to put IDE in the last Amiga models - the A4000 and 1200 - which helped cripple them at a time when they were lagging in overall CPU performance; the man in charge at Commodore the time was the guy that had been responsible for the PCjr disaster..), because it offloaded more logic (and hence was more expensive, hence the cut).

Rather than be exposed to the parallelism, expect that we'll see more higher level functionality being subsumed into peripherals so that the main CPU gets to focus on running your code. E.g. there are TCP offload functionality on some high end network cards.

Just like in the architectures born in the 60's and 70's out of necessity because of wimpy CPUs...

The "one CPU to rule them all" PC is an aberration, born out of a time when CPUs where getting to a performance level where it was possible, and where the norm of single-tasking OSs for personal computers meant it for a short few years seemingly made little difference if the main CPU spent all its time serving disk interrupts while loading stuff.

The norm in computing have been multiple parallel cores. Even multiple parallel CPUs. Often on multiple buses.

And over the last few years we've gone full circle.

If you want to draw one image, GPUs are fine, but they are not really able to simulate neuron networks because they don't have as much parallelism or memory locality a neuron network needs. GPU are more parallized than a CPU, yes, thanks to OpenCL. But they're still specialized towards image blitting, not towards massively parallelism.

Neuron networks are not very hard to understand, but if you think about simulating one, you quickly understand that most hardware is just not adequate and unable to run a neuron network properly, it will be just too slow. Computers were never designed and intended to simulate something like a brain.

Also don't forget most algorithm we use today are not parallelizable, most are sequential. We could rethink many algorithms and adapt them, but parallelism is often a huge constraint. Sequentiality is a specialized case of computability if you think about it.

"[Parallelism] does not necessarily make sense elsewhere. Even in completely new areas that we don't do today because you cant' afford it. If you want to do low-power ubiquotous computer vision etc, I can pretty much guarantee that you're not going to do it with code on a GP CPU. You're likely not even going to do it on a GPU because even that is too expensive (power wise), but with specialized hardware, probably based on some neural network model."

Esp the last bit. I wonder what he means by "specialized hardware, probably based on some neural network model". These ones? http://www.research.ibm.com/cognitive-computing/neurosynapti...

I'm doing Machine Learning / Deep Neural Network research, and of course, parallelism is very important for us. Our chair mostly trains on GPUs at the moment, but the big companies use heavily more parallelism and much bigger sizes, e.g. look at DistBelief from Google.

Parallel sort. Parallel n-nary search. Parallel linear search. A stupid core for the zero-page thread. Hardware blas. Time stepping multi object systems.

i dont know optimum number of cores in mobile cpus. when phone is in your pocket and not used very low power core can get phone going even with bg notifications.