> although it might not be as fast as a hand-written CUDA program

If you're dealing with larger pieces of code at once (that is, if you have a bigger view of what the programmer wants to run, which you should have), one interesting thing you could do is "exploratory compiling". Try different compilation step sequences and transformations, if you have enough invariants in the code. Machine time is cheap and this could yield some interesting results. Of course, the more constrained the language model is, the more opportunities for this.

(There's been some interesting results in this area, like superoptimizers for local optimization, but larger scale compilation would probably use something like genetic algorithms, as you're unlikely to have the opportunity for exhaustive searches.)