What I want is a Linear Algebra interface - As Gilbert Strang taught it. I'll "program" in LinAlg, and a JIT can compile it to whatever wonky way your HW requires.

I'm not willing to even know about the HW at all, the higher level my code the more opportunities for the JIT to optimize my code.

What I really want is something like Mathematica that can JIT to GPU.

As another commenter mentioned all the API's assume you're a discrete GPU off the end of a slow bus, without shared memory. I would kill for an APU that could freely allocate memory for GPU or CPU and change ownership with the speed of a pagefault or kernel transition.