OpenCL was moving to LLVM IR away from programs-as-source-code-strings at some point. Still, you ought to have a JIT step if you want to be able to ship the same binary to multiple systems with GPUs not compatible at the binary level. And BTW shipping LLVM IR on the CPU would make things better in terms of supporting CPUs with incompatible ISAs... whether that ever overtakes native binaries is a question (it did to an extent, of course, with the JVM and what-not.) Of course you could move the JIT off the client device completely and do it at the App Store - prepare a bunch of binaries for all the devices out there - but still, the developers will have shipped IR.

Once you get to shipping IR instead of strings, which is nice I guess in that it should make initialization somewhat faster and the GPU drivers somewhat smaller, I'm not sure what you're going to get from being able to treat a CPU/GPU program as a single whole. Typically the stuff running on the GPU or any other sort of accelerator does not call back into the CPU - these are "leaf functions", optimized as a separate program, pretty much. I guess it'd be nice to be able to optimize a given call to such a function automatically ("here the CPU passes a constant so let's constant-fold all this stuff over here.") The same effect can be achieved today by creating a GPU wrapper code passing the constants and having the CPU call that, avoiding this doesn't sound like a huge deal. Other than that, what big improvement opportunities am I missing due to the CPU compiler not caring what the GPU compiler is doing and what code the GPU is running?

(Not a rhetorical question, I expect to be missing something; I work on accelerator design but I haven't ever thought very deeply about an integrated CPU+accelerator compiler, except for a high-level language producing code for both the CPU and some accelerators - but this is a different situation, and there you don't care what say OpenGL or OpenCL do, you generate both and you're the compiler and you use whatever opportunities for program analysis that your higher level language gives you. Here I think the point was that we miss opportunities due to not having a single compiler analyzing our C/C++ CPU code and our shader/OpenCL/... code as a single whole - it's this area where I don't see a lot of missed opportunities and asking where I'm wrong.)