SLURM is mentioned by @momofuku.
Ray is another good candidate as well (and feels more modern imho).
Ray is another good candidate as well (and feels more modern imho).
EDIT: so the main thing is technologies like Ray have a way to do these things, but I honestly just want an easy way to do this. Maybe means I will have to set up something with Ray and AWS myself and make a wrapper for that?
https://docs.ray.io/en/latest/ray-core/key-concepts.html#tas...
https://docs.ray.io/en/latest/ray-core/examples/gentle_walkt...
That's what I would want something to do if I was building tooling like this myself. Although, I'd do it in golang instead of python so that the dependency chain was simpler. A single small binary is nicer imho.