​Supercomputer leaders come together on new open-source framework

If you have a job and it just needs one core (your code is embarrassingly parallel), that's all well and fine, except you can't just get one core, you get 16 or 24 as the smallest unit. That's fine, except now you end up writing a task queue, and a system which spins up new workers whenever the old ones are killed due to wall clock time. But most of these supercomputing centers aren't too keen on having services running locally, so now your task queue is a remote system, and now you've got to do clever things wherever you can to make all this work the way you want it to (leverage any remote APIs, write SSH tunnel wrappers, etc...) Now you've got another batch system for a batch system.

The solution to this at the system level is to instead also provide serial queues which schedule one job to one core. But if too many serial jobs are in the works, especially if they aren't confined to a subset of the nodes, the true MPI jobs which will struggle to get scheduled. This isn't too much of an issue so long as everyone has a wall clock, but it's definitely not ideal.

Ideally, you'd provision your machines, run a few services, and have them as long as you need them, do a bunch of shit, and release your machines. But it doesn't work that way.

More pain comes around should you need to stage in/out your massive amounts of data.