GCP Instances that are scaling up and down based on usage. Lot of image freezing, for faster boots, etc.

One hack is for the medium-level models, you can actually run them on Cascade Lake (which are sort-of more optimized for ML) than traditional processors. There's a 30% performance there. Mathematically, GPU vs CPU in performance at inference time, you're paying an annoying premium!

Right now, the default writing style "medium" is running in Cascade Lake (no gpu). I also (over)optimized the microservices running the endpoints too.