I think time-slicing for GPUs is likely the solution here.

If you could checkpoint a GPU quickly enough it would be possible to run multiple isolated workloads on the same GPUs without any issues.