Cramming an LLM platform into one container

Getting this to work was an adventure. I had two experiences that shaped this.

* One was building an LLM optimized training datacenter at Baidu.

* The other was deploying software as a medical device in three US hospital networks.

The former pushes the design towards bare metal and speed of light performance. SLURM, MPI, GPUs, mixed precision, all reduce, data parallelism, model parallelism, memory efficient optimization, compressed communication.

The latter pushes it towards complete encapsulation, OCI, air gapped deployments, and no dependencies.

I suspect that putting them together is something new, but I'm curious to hear what others are doing.

What is your favorite LLM stack? Why do you love it? Or why do you hate it?

I also have some points about existing systems that I hate:

One issue I've had with building everything on top of web scale infrastructure is that it isn't optimized for speed/locality and scaled out fault tolerance in the same way that an HPC cluster is.

Another issue is that these platforms are usually supported by big infrastructure teams, which assume a lot of operational support to do anything. I can't just boot it on my laptop to get a dev environment.

I can't deploy it over an air gap either because the setup/ops is so finicky and manual.