A critical difference is that GPU architectures differ far more, AND different GPUs (well, their driver's) effectively use different compilers with different optimizations, etc. (Yes, the input is an IL with some optimizations already done, but ultimately codegen is done by the driver.)

You can micro-optimize for a particular CPU- say, not using AVX-512 on Zen, or hand-placing instructions so the latencies all line up juuust right- but it's not nearly as important as for a GPU.

Plus, GPU vendors will work with software devs to optimize the driver just for that software (mostly via a collection of weird workarounds, hacks, feature flags, etc. in the driver, automatically applied when it detects that software running); often the software dev will take, essentially, a bribe from a GPU manufacturer to only work with them.

Then you have stuff like PhysX, CuDNN, etc...