There's some really early days work on making training loops robust to failure but they all have trade-offs right now.

I remain hopeful that we'll be able to democratize the entire tech stack for this tech.