Thanks for asking, thats a good point! Compared to spot/preemptible, your files will be persistent. You can stop your running pod, add more GPUs and then continue training your model.