Hacker News Reader

Comment by papichulo2023 | Hacker News Reader

papichulo2023·

I was under the impresion that modern AI still runs mainly baremetal. Isn't pytorch parallel distribute not good enough?

View on HN

You can use kubernetes to manage bare metal machines, but it probably doesn't make much difference: once the bits are on the GPU, I doubt a properly configured VM is getting in the way, let alone containerd.

the problem is multinode runs that communicate through the network

Yes, which is especially important for training. Getting good GPU interconnect can be really important for training large models.

Multinode runs don’t communicate through the network in a DGX configuration. NVlink allows for RDMA over direct infiniband. No need for network here.

Infiniband is a network too…

But even if we set that aside you’ll get access to your data over a network connection because these are expensive nodes running batch jobs with finite disk space, not personal workstations.

Yea ofc. Nvidia has for mellanox infiniband, nvlink external, and even mellanox ethernet pci-to-pci, no need to involve the CPU. nvidia-docker has a few mods to support this too.

There are a ton of tricks you can do to share hbm/flops amongst GPUs in a cluster.

> I was under the impresion that modern AI still runs mainly baremetal.

Generally yes, but you can do (PCI) pass-through of GPUs to VMs and containers and such, so it is not strictly necessary.

You can give GPU to VM and containers without any overhead these days. However, Nvidia always placed roadblocks when you aren't using their data center hardware in datacenters.

This is purely a scheduler, it doesn't have any any effect on the workload performance.

nVidia Multi-Instance GPUs has been around for >4 years.

Reply on news.ycombinator.com