This paper from Microsoft [1] is the coolest thing I've seen in this space. Basically workload (deep learning in this case) level optimization to allow jobs to be resized and preempted.

[1] https://arxiv.org/pdf/2202.07848.pdf