The cheapest cards that will work are in the $100 range, hardly a major expense.
The GPGPU is probably one of the biggest developments in the last couple of years, it puts enormous power in the hands of individuals with a surprisingly small power bill and footprint.
~300W for a 480 core machine is not bad at all!
I'm not sure if most people understand the kinds of threads/cores that GPUs actually have, and with that, the kinds of computations they're actually good at. A GPU's threads are extremely simple execution pipelines - nothing like the cores of an Intel Quad Core chip. Groups of GPU threads actually execute together in lock-step, executing the same code (but with different parameters).
Such devices are extremely good at handling data parallelism - where the same computation is performed on a large chunk of data - seen in areas such as graphics and scientific computing. But you're probably not going to, say, speed up your webserver with a GPU. If you don't have a computation that is extremely data parallel - that is, capable of having thousands of execution contexts working on data at the same time - a GPU won't help.
A GPU won't help with our current programming model, maybe there is room for a new model then ?
That won't happen overnight, but one thing is for sure, if you want to get more performance than what your applications are giving you today then you'd better learn how to code for massively parallel systems.
Certainly, the computing world is going parallel. My personal belief is that what we end up with will be a heterogeneous chip with some parts good at massively parallel computations, and some good at sequential computations.
But I think GPUs themselves represent a stop-gap; the communication cost of transferring data off the motherboard to the card is just too high for fine-grained parallelism. Once we have that sort of architecture on-chip the game changes. Intel's Larabee and AMD's Fusion promise to do this. The Cell was close, but making the vector processors completely divorced from the normal memory hierarchy makes programming for them much more difficult. (Although it does give me many research opportunities.)
I agree that GPUs are a stopgap, but they're a pretty interesting one.
I've had all kinds of co processors over the years, small clusters, DSP daughter boards, transputers and so on. Nothing to date comes even close to the power of the GPU boards that you can buy right now.
The comms cost is indeed where the problems lie, but that can go two ways:
1) a more formalized form of high speed interconnect (such as infiniband directly on the GPU card)
2) a port of a general purpose OS to a GPU.
Some people are already working on (2), (1) is probably not in the cards unless somebody orders a very large number of boards.
There are motherboards that will hold 4 or even 6 PCI express cards too.
I don't have any experience with the cell architecture, I'll go do some reading, it sounded interesting enough when it was released but I never got around to study it. More reading to do I guess :)
Personally, I think that GPUs as we know them won't be around long enough for 1) or 2). Moving the GPU (or a GPU-like computation accelerator) onto the chip itself changes everything.
I think it's also worth keeping in mind that GPUs are, essentially, tiny computers unto themselves. Putting a GPU in a system now is like installing a computer in a computer.
I've been doing exactly that for the longest time, first with a DSP32, then with a transputer card. Then I switched to small scale clusters (the largest had 10 nodes, see here http://www.clustercompute.com/ ), now GPUs.
Your take on 1) and 2) is noted, thanks for that insight.
Researchers (especially in parallel computing) are already figuring out how to port all sorts of algorithms over to the GPU -- and not just scientific or graphics code -- we're talking general purpose algorithms like sorting, ranking, cryptography, networking, database operations, etc:
http://gpgpu.org/2009/04/13/efficient-acceleration-of-asymme...
http://gpgpu.org/2009/04/28/fast-and-scalable-list-ranking-o...
http://gpgpu.org/2008/12/11/wait-free-programming-for-genera...
http://gpgpu.org/2008/10/26/gnort-high-performance-network-i...
http://gpgpu.org/2008/05/25/a-fast-similarity-join-algorithm...
The biggest obstacle to preventing widespread adoption of GPGPU techniques today is that things are still in so much flux that mature, stable APIs have not yet emerged. There is fierce competition between Intel (Larrabee), Apple/AMD (Grand Central/OpenCL), and NVIDIA (CUDA) to get their respective APIs to be the dominant ones, and no one's yet come up with a mature wrapper API that can target any of them.
This is in large part because the hardware itself is still in flux, morphing away from the graphics-specific pipeline of 10 years ago and into a general purpose SIMD architecture. This has meant a change in the kinds of operations allowed on the GPU. For example, branches used to be a big no-no in graphics hardware: the performance hit caused by a branch was disastrous on a massively-multicore system. However, all hardware makers are now adding supporting for branches, because it's almost impossible to port a lot of CPU algorithms without them.
In any case, I'm confident that within a year or two, things will have settled down somewhat, at which point there's going to be a mad dash by developers to start using GPUs (probably systems- and library-level developers rather than application-programmers).
Stuff like this is still cutting edge but will be mainstream soon, the time to gain experience is today. Not to wait.
The longer that time the better suited the problem is for a massive parallel solution.
If the time is low relative to the IO that needs to be done then you'll find very soon that the bus that carries data between the host CPU and the number cruncher is the bottle-neck.
Still, PCIe 2.0 x16 is limited to 8GByte/s so I guess the real question is how many matrixes where you multiplying?
I'll defer to your judgment about this stuff, then. (I'm a PhD student in computer vision, and we've recently been using a GPU SVM library that's been amazing for cutting down our processing times, so I guess I've been a little dazzled by this stuff.)
Anyways, since you're in this field, what's your feeling about the future of parallel computing, with regards to the different vendors? Which of CUDA/OpenCL/Larrabee will win out? Or none of the above? When will APIs settle down?
Your question is the question in parallel computing right now. And it effects all sizes and scales, from processor architecture (look at the different architectures of an Intel Quad Core, Cell, GPUs and upcoming Larabee and Fusion) to supercomputers (BlueGene style thousands of slow cores with fast interconnect, RoadRunner style of typical multicore processors with Cells as accelerators, Nvidia's giant GPU box, or just lots of SMPs). We don't know what the future will look, which makes this an interesting time to be in the field. People at all levels are experimenting with different architectures. We don't know what will win, if any one thing will win, and when we'll know.
With that said, I don't think APIs at the processor level will settle down until the hardware does. My understanding of OpenCL is to have a programming model that would work on architectures as different as GPUs, Cell and Larrabee, and that this will supplant Cuda. That sounds like a great idea, but lots of great ideas haven't worked in practice before.
I think it's going to be several at least several years of experimentation before the hardware settles down. My own belief (that is, opinion not based on experimental data) is that we'll end up with a heterogeneous chip with lots of simple cores for parallelism, a small number of sophisticated cores for sequential computation, all part of an integrated memory hierarchy.
a) Explicit data copy required to and from GPU makes GPGPU useless for many problems.
b) GPUs are not as general purpose as CPUs. For example, no indirect jumps (goto address x) is allowed so no real subroutine calls. And no recursion.
c) GPUs are not good at branching. The toll on branching can be heavy.
d) Caches are fairly small. The on-chip shared memory, which is essentially a software controlled cache is a good idea in some cases but its still of a very small size.
Well the thing is that the computational peak power of the GPUs is not free. It comes at the cost of reduced flexibility. Nothing is free in computer architecture.
Use this new technology and your app will be faster - but will only work if your customers have one of this range of graphics cards.
Your customers will get a huge speedup if they have a supported card, but it will still work if they don't.
It's like the software floating point emulators back in the 386 days