> I fail to see how any sort of much lower level access to the machine would be portable across price ranges and microarchitecture generations

That's the assumption that can be removed. GPUs don't have stable ISAs, and their assembly-like code gets recompiled for each microarchitecture.

In the Intel's world of prebaked machine code adoption of a wider set of SIMD instructions takes a decade+. In GPUs it's just a driver update.