DLRover: Distributed Deep Learning System for LLM Training

  DLRover makes the distributed training of large AI models easy, stable, fast and green

  DLRover can restore the training when the process fails without stopping the training job.

  In addition to fault tolerance, DLRover provides the flash checkpoint to save/load checkpoint in seconds.

I've personally never trained a model large enough to warrant the use of tools like DLRover, but I definetly see the intended usecase. I do however wonder if re-scheduling a task that failed due to OOM (one of the provided examples) won't just fail again due to OOM on another node.

I'm a stickler for using correct terms, one nitpick I have is the "green" descriptor. The repo does not ellaborate on how DLRover makes the process more "green", but I can only assume that they mean it helps with resource management, which in turn could make the process more energy efficient. If that is true, the authors might consider replacing "green" with "resource efficient".