Jim Keller suggests Nvidia should have used Ethernet to link Blackwell GPUs
tomshardware.com
tomshardware.com
But why would NVIDIA do anything to share this moat? Or even risk to have their whole ecosystem go glitchy and buggy with other hardware vendors? They invested heavily, very heavily, in network tech and did the whole integration (nccl, gpudirect, and even simple memcopy works fine with NVLINK without futzing with actual Ethernet/IP stacks) at a moment where everyone was fucking around either building GPUs back from scratch or buying and dumping good tech? If there was an actually competitive alternative, I could see Jensen have a look but I don't see it right now.
> "The reason why we took [NVLink] out was because we needed the we needed the I/Os for something else, and so, so we use the I/O area to cram in as much as much AI processing as we could," Huang confirmed and explained of NVLink's absence.
> For gamers, creators, and workstation users who want a multi-GPU setup with the latest RTX 4000 series, Nvidia is switching from NVLink as a bridge to the PCIe Gen 5 standard.
PCIe 5.0 x32 supports 128GB/s.
"PCIe 7.0 Draft 0.5 Spec: 512 GB/s over PCIe x16 On Track For 2025" https://news.ycombinator.com/item?id=39933959
> The HBM3E [RAM] Wikipedia article says 1.2TB/s (2024) https://news.ycombinator.com/item?id=39938652
"Fiber-optic data transfer speeds hit a rapid 301 Tbps" (2024-03) https://news.ycombinator.com/item?id=39864107
HPE Cray Slingshot is a copper Ethernet HPC interconnect: https://www.nextplatform.com/2022/01/31/crays-slingshot-inte... ... "HPE Slingshot Makes The GPUs Do Control Plane Compute" https://www.nextplatform.com/2022/08/15/hpe-slingshot-makes-... :
> Here’s the gist. With normal GPU aware MPI software, inside the node the GPU vendors – the only two that matter today are Nvidia and AMD – have their own mechanisms for doing peer-to-peer communications between the GPUs – in this case NVLink and Infinity Fabric. For MPI data exchanges between GPUs located in different nodes, which is often required for running large simulations and models as well as for large AI training runs, MPI data movement is done through Remote Direct Memory Access methods, pioneered with InfiniBand adapters decades ago, which allow for the transfer of data between a GPU and a network interface card without the interaction of the host CPU’s network stack. [...]
> In a [recent paper], HPE showed off the new MPI offload method, making the distinction between the typical MPI communication that is GPU aware versus the stream triggered method that is GPU stream aware.
"Exploring GPU Stream-Aware Message Passing using Triggered Operations" (2022) https://arxiv.org/abs/2208.04817 https://scholar.google.com/scholar?cites=1354761056738606744...
The theory sounds nice, but isn't the RTX 4090 only PCIe gen 4, not 5?
* Fiber seems... overpriced. I'm not quite sure why. USB TOSlink is $3 on Aliexpress, as is the cable, so there's nothing fundamentally expensive there, except for being enterprisey. A lot more interconnect would make sense to have as optical.
* Wifi is increasing in speed. Wifi 8 is planned to hit 100Gbps. That's where the investment seems to be going.
Ethernet has issues too, at times. I've certainly seen nominal 10Gbps networks degrade to <1Gbps, and to fix that, you need to debug wires and connectors, and see where the signal integrity issue too. As speeds go up, those issues get more common. 10Mbps, you could almost literally run a pair of wires from Radio Shack and it would work. 1Gbps was the breakpoint where if I didn't use proper everything (which is often mislabelled on Amazon / Aliexpress / eBay / etc.), things would degrade.
On the other hand, my wifi has a nice MIMO router.
I expect that split will continue. In theory, a MIMO router with an array of antennas could be almost point-to-point, directing signals where they need to go. At the same time, as speeds go up, any parasitic cable / connector inductance or capacitance will have increasing impact.
Newer wifi speeds are nifty for general consumers, but a lot of people (prosumers, small business) have moved on from 1GbE wired networking to much faster things. There's more than just 10GbE available (cheaply too if 2nd hand is ok).
Wired networking also lets multiple people use the network at the full speed too, and tends to be a reliable experience. Well, more reliable than wifi anyway. ;)
In theory, MIMO and antenna arrays do too. Practice isn't quite there yet, but it's getting there.
In general cables are much better if you can use them. The fundamental principle of EM waves is the same, but you get to decide whether these waves are inside a tight strand of material or just all around.
"I don't have a horse in this race, but I saw today that there was a healthy thread <https://news.ycombinator.com/item?id=40011762> about "just pull fiber" when folks were discussing pulling cat6e and a lot of that talk was about the relative cost versus massive bandwidth gains."
For home use, ethernet has increasingly moved into a hassle / performance point where I'm better off with either wifi or, if I wanted faster speed, fiber. I just wish fiber was cheaper.
I’m confused by your whole sentiment. They said use fiber instead of wiring for 10Gb ethernet, and you’re just agreeing but naming alternatives that underperform it?
It’s not very much more expensive to get a couple fiber NICs.
I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency.
What's the alternative(s) you're thinking of?
There are some specialist things inside supercomputers (in the cases that Infiniband isn't used), but there doesn't seem to be anything else reasonably available commercially.
Omni-path (a descendant of Infiniband) doesn't seem to have worked out either.
---
Infiniband also has advantages over Ethernet in other things, like bundling links together.
With (standardised) ethernet, adding more cables/links between two hosts gives more overall bandwidth in total. But the additional links can't be used to increase the bandwidth of a single stream.
With Infiniband, when you add links they just combine together into a single bigger pipe, which can transfer more traffic in a single stream. Less need to jump up to higher tier adapters and switches with IB for that, you can just run more links between the hosts.
"I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency. "
I don't agree with Jim Keller who says that Ethernet would be as good or better, in that when you use Infiniband, you get the absolute highest performance from IB, since IB does best on bandwidth and latency considerations.
A super common example of this is when consumers want to upgrade their existing home infrastructure to use a new NAS, and they start hitting the limits of 1GbE. Their first question is often along the lines of "can't I just add more links together to give more bandwidth?".
Then people need to explain "No, ethernet doesn't work that way. You need to move up to 2.5/5/10GbE". (etc)
Infiniband doesn't have that problem.
A single stream would have its frames striped over all the links in the bond, due to the configuration of the sending host. The flows would remain parallel and arrive over N switches to N NIC ports on the destination host. Care had to be taken with topology to keep parallel paths from end to end such as in one datacenter. Generally, flows could fan-in somewhere on the path and then go over a higher speed uplink, but would never fan-out again to parallel links.
The drawback is more frame (packet) reordering, so it is even more sensitive to buffering and congestion control effects. If there were switch failures, the flows could become unbalanced if the topology offered redundant cross-links, since this would cause some fan-in on the surviving switches. Otherwise, completely independent switch stacks could be used, so a failure means traffic is dropped. Then, the host needs to do some kind of remote peer connectivity check to decide whether each link is healthy or not, and drop unhealthy links from the bond.
Ironically, this could be done in consumer-grade LANs, since it does not depend on bonding features of the switches. However, careful setup is needed with consideration of the topology and switch behavior, to avoid pathology.