I'll defer to your judgment about this stuff, then. (I'm a PhD student in computer vision, and we've recently been using a GPU SVM library that's been amazing for cutting down our processing times, so I guess I've been a little dazzled by this stuff.)
Anyways, since you're in this field, what's your feeling about the future of parallel computing, with regards to the different vendors? Which of CUDA/OpenCL/Larrabee will win out? Or none of the above? When will APIs settle down?
Your question is the question in parallel computing right now. And it effects all sizes and scales, from processor architecture (look at the different architectures of an Intel Quad Core, Cell, GPUs and upcoming Larabee and Fusion) to supercomputers (BlueGene style thousands of slow cores with fast interconnect, RoadRunner style of typical multicore processors with Cells as accelerators, Nvidia's giant GPU box, or just lots of SMPs). We don't know what the future will look, which makes this an interesting time to be in the field. People at all levels are experimenting with different architectures. We don't know what will win, if any one thing will win, and when we'll know.
With that said, I don't think APIs at the processor level will settle down until the hardware does. My understanding of OpenCL is to have a programming model that would work on architectures as different as GPUs, Cell and Larrabee, and that this will supplant Cuda. That sounds like a great idea, but lots of great ideas haven't worked in practice before.
I think it's going to be several at least several years of experimentation before the hardware settles down. My own belief (that is, opinion not based on experimental data) is that we'll end up with a heterogeneous chip with lots of simple cores for parallelism, a small number of sophisticated cores for sequential computation, all part of an integrated memory hierarchy.
The longer that time the better suited the problem is for a massive parallel solution.
If the time is low relative to the IO that needs to be done then you'll find very soon that the bus that carries data between the host CPU and the number cruncher is the bottle-neck.
Still, PCIe 2.0 x16 is limited to 8GByte/s so I guess the real question is how many matrixes where you multiplying?