Some interesting things to note, at least from my point of view:
* GPUs should not be considered like the next generation CPUs at this stage.
* Trying to take advantage of CUDA/OpenCL not only requires redesigning your algorithm and altering your data structures, but also the folly of implementing in software what is already available in hardware.
* Thread shared memory on nVidia cards isn't exactly similar to a cache. There's also a lot of papers in the last two years that speak of nothing else but altering algorithms to make better use of CUDA, because for anything else but the simplest of raytracers on a few objects, the situation gets really bad.
Very nice post, though. Neatly organized and well written, I really enjoyed it.