Hacker News Reader

Comment by eigenvalue | Hacker News Reader

Lots of good and detailed information here, thanks. I'm curious why Ethernet interconnect is so unreliable in practice compared to the Infiniband. I would think that at this point, after a decade or more of current Ethernet standards, all the kinks would be worked out and the worst that would happen would be occasional latency spikes and a few lost packets that could be retransmitted quickly. Shouldn't the training frameworks be more robust to that sort of thing?

View on HN

Infiniband and ethernet are very different at the lowest levels. Ethernet interconnects use RoCE (RDMA over converged ethernet), which actually encapsulated infiniband transport in ethernet, but you still pay for higher routing latency, and you need separate compute-network and storage-network to avoid queueing (lossless ethernet).

https://community.fs.com/article/infiniband-vs-ethernet-whic...

https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet

Also... don't underestimate the PCI bus bottlenecks when you put 8x 400GB networking + 8x GPUs. There are ways now to have a tree of PCI switches and avoid overloading the main one, each GPU gets its own networking card and PCI switch.

This is a great comment.

Our cluster is 128 GPUs into a single Dell switch... should help with the queuing. We also have a separate e-w 100G network.

This is why we went with Dell XE9680 chassis... people forget that PCI switches are quite important with this level of compute. Dell has done a good job here.

Interesting, thanks. From the wikipedia link, this seems like the probable culprit for why things break:

"Although in general the delivery order of UDP packets is not guaranteed, the RoCEv2 specification requires that packets with the same UDP source port and the same destination address must not be reordered."

In practice it's easy to design an Ethernet network that doesn't reorder packets.

But if packets have to go over the internet at some point, aren't all bets off if you use UDP?

ROCE doesn't go over the Internet, most ISPs don't reorder packets, and UDP isn't treated specially.

RoCE does not encapsulate infiniband

You are right. Sorry, I quoted the linked article. I haven't worked on the networking side to that level of detail.

ML training is a tightly interconnected HPC style workload (network bound), which is a market Infiniband has been targeting for a long time.

Nvidia saw that and bought Mellanox, and made NCCL/GPU's work really well with Infiniband.

Large public clouds already have huge investments in ethernet and don't want to be further locked into Nvidia, so Nvidia does have a roadmap for ethernet GPU clusters (roughly 1 year behind Infiniband).

But if you are building your own Nvidia cluster, it would be silly to build it on ethernet. Just buy exactly what Nvidia recommends, you are already locked in anyway.

Reply on news.ycombinator.com