Upgrading Multi-GPU Interconnectivity with the Third-Generation Nvidia NVSwitch | Hacker News Reader

Upgrading Multi-GPU Interconnectivity with the Third-Generation Nvidia NVSwitch

developer.nvidia.com

48 pointsmy12321 comments

RDNA3 has memory/cache chiplets providing 3.5Tbps. The 0.9Tbps of much further distance chip to chip conmevtivity here is highly impressive. Interesting to note that RDNA3 links are only 10Gbps, they just have extreme interposer design with hundreds upon hundreds of incredibly small wires (resulting in stupendously high bit-efficiency); not something a multi-GPU design can do, and hence Nvidia doing 100Gbps links.

Given where ethernet is (on the way to terabit ethernet) Im not surprised Nvidia is here, but it's still cool to see.

The onboard SHARP processor on the NVswitch is very cool to read about. Given how much ML involves crunching many matrixes/tensors then adding the results, being able to parallelize the crunching and have a central aggregator makes sense. In general I hope we see an emerging presence of extremely high-fabric connectivity cpus & gpus that can serve as DPUs like this, as central coordinators. Thinking of this less as a switch (which it certainly is too, with it's vast bank of PHYs and massive crossbar) but more as an io cemtric computer is, I think, deserved.

Can anyone elaborate on the pros and cons of nvlink in comparison to e.g. rdma over infiniband/roce, or pci-e?

According to the article, it has more than three times the bandwidth of PCI-E Gen5 per lane.

I think NVLink is a memory interconnect more like NUMAlink, UPI, or Infinity Fabric. AFAIK GPUs can transparently access each other's memory over NVLink (probably at cache line granularity) while RDMA protocols require explicitly copying memory (usually in larger chunks) before accessing it.

it's more than 6x the bandwidth of practical network speeds today.

I would have compared with PCIe bandwidth, which seem to drag behind actual network speeds these days. In the PCIe 3 era, Mellanox/NVIDIA sold (still sell) 200g (2x100 mostly) boards using 2 PCIe3 16x ports, linked by a specific cable (the whole montage is called SocketDirect? or sometimes PCIe 32x?).

When PCIe 4 started getting standardized you could already saturate the bus with one 200G NIC. Now ConnectX 7 is out, should be able to saturate PCIe 5 but most constructors haven't got their motherboards out. SFP112 is out and you can already saturate PCIe 6 with 2 QSFP112 (or equivalent 2x4x100G). Still need CPUs with support or GPUs with PCIe5 or 6 support, which I guess are coming or you can stream from one NIC to several GPUs.

It might be more interesting for nvidia in the mid to long run to not wait for PCIe standards to settle and to allow streaming to/from their NICs to their GPUs (or CPUs, see Grace Hopper white paper) using nvlink, never touching the PCIe bus...

the problem is PCIe speeds are increasing faster than network. pcie5 can hit 500Gbps in theory, but the network is only 400Gbps.

Has anyone used one of the Nvlink switches? That must be a power and heat beast.

Are those the things that kept melting and blowing up computers with the 4090s?

No. Some 4090s had issues with power connectors. The 4090 does not support NVLink (though there is evidence that they were designed with the capability).

Probably they are using the same PCBs as their Datacenter counterparts and there NVLink is extremely important to minimize overhead.

They're not using the same PCBs because 4090 has the blow-through cooler and so a downsized pcb but the DC cards have big fanless heatsinks for cooling by chassis fans.

3090 had nvlink so it was probably considered for 4090 but eventually cut for some reason

I imagine the reason was that hardly anyone used it, and those that did use it were the same group of clients who would traditionally buy Quadro cards.

Oh yeah, the artificial market segmentation.

Is there natural market segmentation as well?

The AD102 die this gen doesn’t have NVLink on silicon. RTX 6000 Ada Generation doesn’t have NVLink, and neither does the L40.

Is the white paper out and precise specs available for L40? Last time I looked nvlink was still with a question mark on L40? Doesn't make sense to remove the nvlink feature from the A40 line that had some nvlink, that would piss off a bunch of users hoping to upgrade to not-H100...

Entertainingly, they do use molex micro fit connectors on SXM boards - the same ones as the 4090.

Your question is understandable given the hysteria about those connectors, but they're power connectors and it seems there were only ~50 cases in total, mostly down to the connector not being inserted fully (with a handful due to manufacturing defects).

It happens

NVSwitch looks around 1/4th the size of a contemporary Ethernet switch so it may be only ~100 W which is pretty small relative to the GPUs it's connecting.

Reply on news.ycombinator.com