Nothing fancy. The core principles are the same, you'll need to adapt them depending upon the kind workload changes that ML introduces. For most ML systems 1. Storage infra: assuming storage for models or even data, use any blob storage like S3. Or a shared networked file system like EFS, Lustre etc. 2. Networking: if you're talking about networking using large GPUs, I am not aware of any definitive resource on this. 3. Scheduling: This is honestly a solved problem at this point, anything works - write your own coordinator that periodically runs docker image base jobs (you can hook one up quite quickly using some sort of system for metadata and triggers powered by message queuing), use Airflow, use AWS Batch for large scale jobs.

You missed model serving (I think ?). Tough and latency sensitive esp for recommender systems. Prone to latency spikes, traffic spikes. Even with a well-written Python code you can run into limitations quite quickly.