Migrating from Slurm to Kubernetes

Much of this stems from Google's original design for its search engine, and borg. Borg was not a batch scheduler, it was a service-keeper-upper with loose relationships between individual containers. Google's design was never friendly to MPI-style jobs (which long ago tended to be highly synchronous, with distributed processes that were extremely long-lived, and servers did not just "come and go").

There are a number of properties of SLURM and other batch systems that are far more convenient for users. Typically a SLURM system will have a large distributed filesystem that can be accessed from all the nodes using normal UNIX commands (GFS and Colossus aren't mountable filesystems and the UNIX tools do not really work against them natively). Reading output logs is often much easier (tail -f, less, etc) but the cost of the filesystem (per byte) goes up exponentially as the number of nodes increases.

I have extensive experience with both systems. My experience with SLURM is that the system is highly predictable, slowly changing, and I can get my work done. Whereas on Google systems, everything was breaking constantly and I had to focus on making my systems resilient to noise. On the other hand, the Google system scaled far larger, for cheaper.

When I joined Google I asked Jeff Dean how page rank was computed and he said it was an iterative mapreduce- I was assuming it was some sort of tightly coupled supercomputer-style job, but for the size of the web, and the link structure, MR made much more sense.

The author doesn't seem to now even the basics of SLURM?

"gang scheduling" according to the official docs: https://slurm.schedmd.com/gang_scheduling.html

-- maybe I've read the docs wrong the last decade using SLURM.