Who Wants Parallel Computers?
rjlipton.wordpress.com
rjlipton.wordpress.com
I think almost everyone involved in the industry knows that we want faster uni-processors, and that we don't want multi-core machines. We're pretty sure that some problems that do have fast serial algorithms don't have efficient parallel algorithms (this is akin to P vs NP type questions, and is beyond my current knowledge) and these problems are stalled at uni-processor speeds.
Research continues, progress is made, and most people don't care because most people just write emails, use facebook and browse the web. Heavy computing with parallel or multi-processor machines is hard in general, and that's not changing fast.
I don't expect it to. People will simply have to learn how to devise parallel algorithms, and write good, parallel code.
In the last couple of years, we have seen the rise of the netbook, the smartphone, and languages like Python and Ruby. Each of these are, apparently, "fast enough" to be highly succesful.
Applications are also changing: casual games, Farmville and even World of Warcraft have very modest system requirements but make a ton of money. Web applications are awful from a performance perspective: they are crippled by network latency and need a single server to do the work for many people. Still, they are "the modern way" to write an application. Again, the performance is "good enough".
Finally, even when performance is needed, there are different models for parallel programming than multicore. GPUs are very fast data-parallel "single core"-like devices. At the other end of the spectrum, real-world "web scale" systems scale horizontally and are thereby forced to adopt a multi-process model.
Yes, our OSes should be written for multicore, the newest Doom/Quake should be written for multicore, and our numerical models should be at least aware of multicore; but despite the fact that the C programmer in me will likely be happier in your multicore world than in the world I sketch above, I think most programmers will live in the latter.
I don't think that's accurate - being "single core"-like. Besides literally having multiple execution cores, GPUs are radically data-parallel in ways that even those already familiar with data-parallel code (using languages such as OpenMP) must still adapt their thinking. Code that extracts high performance out of GPUs must be aware of the memory hierarchy and degree of parallelism - you can't pretend you're doing sequential programming.
But I agree, "single core" isn't really true.
These days I think energy usage is the overriding concern, both because of thermal limits in high performance systems, and energy budgets in low-power systems. A smartphone which is fast enough to play Farmville or run a python interpreter is cool and sells well, but there is still a huge economic incentive to cut energy usage further, since that would increase battery life.
The core of your argument seems to be that since things like netbooks, smartphones, Python and Ruby are currently "fast enough", there is little incentive to increase the core count. I disagree with that, because energy savings will (or would) still add massive economic value. Whether these kinds of systems will become reality is a different question, since of course programming them is a very hard problem.
But how cool would it be to have a smartphone with 100 cores, each running at a few MHz or so (and apps which use them efficiently) and battery life of a month or so? Also, think of how motivated companies with datacenters are to reduce energy costs.
For consumer purposes. However, a ginormous amount of the software in operation today is not aimed at consumers at all, and don't resemble Farmville in the slightest.
Most programmers are writing COBOL (even if they are using a slightly more modern language to do it in.)
If the above is not an answer to what you meant, could you please clarify?
The tool I have now allows people to write multi-process programs and have them communicate effectively. I'm still trying to get permission to release the system as open source, or at least publish the ideas, but the industry is notoriously secretive and conservative, so it'll be some time yet.
Moore's Law still holds; it is about the number of transistors on a chip, not processor frequency. The problem is that the way that we achieved better performance for a long time was to increase sequential performance, which meant: higher clock frequency, larger caches, longer and more elaborate instruction pipelines to extract instruction level parallelism. That's what is breaking down, not Moore's Law.
That we're now exploring multicore design is because of Moore's Law: our old approaches for improving performance no longer work, but we're still getting more transistors to work with.
If it were economical to produce faster uniprocessors instead, you can bet that chip companies would do it, because they'd make more money that way.
Like Matt Welsh(the Harvard prof who left for Google) mentioned on his blog recently.. how many people risk running OpenMP in production, and watch some nodes fail..
Even the "Big" guys (Google,Facebook) seem to be risk-aversed to thread-level parallelism to earn their bread -BigTable,Hadoop,Cassandra are centered on multi- process architecture.. and basically built for horizontal scaling- add more nodes,assume they fail.
Where is this push towards parallel machines coming from? Not really from consumer web - Or any consumer oriented service that an entrepreneur can gain value, by utilizing multi-processor systems.
It's the organizations with big bucks that call the shots, or drive innovation. Like NCEP/NOAA in weather prediction. Not the guy who moves to the Bay area who wants to hack something, get traction in user base, get funding,and finally mint money :-)
The demand for writing consumer facing software for multi-processor systems is simply too less now.
Brains are big parallel machines and it seems inevitable that computers also follow the path to massive parallelism that evolution took long ago.