Something like global optimization has been done in pytorch, here's a blog about it: https://www.photoroom.com/tech/stable-diffusion-25-percent-f...
Mixed precision seems pretty much default looking at a few Stable Diffusion notebooks.
More intriguing, there's also a more local optimization that makes pytorch faster: https://www.photoroom.com/tech/stable-diffusion-100-percent-...
Unless it's already there, that last one would be interesting to add to keras.
All in all this machine learning ecosystem is wild, as a software dev, things like cache locality and preferring computation over memory access are basic optimizations, yet in machine learning it seems wildly disregarded, I've seen models happily swapping between gpu and system memory to do numpy calculations.
Hopefully stable diffusion changes things, the work towards optimizations is there, it just seems often disregarded. As stable diffusion is one popular open model that, when optimized, can be run locally (and not as saas, where you just add extra compute power, which seems cheaper than engineers) and has a lot of enthusiasm behind it, it might just be the spark that makes optimization sexy again.