There is an excellent paper and talk on how Google's TPU cluster is managed: https://www.usenix.org/conference/nsdi24/presentation/zu.