Hacker News Reader

Comment by teraflop | Hacker News Reader

Ethernet flow control doesn't really fix congestion except in very special cases.

Consider a very simple topology:

    A         C
     \       /
      S1===S2
     /       \
    B         D

Say hosts A and B are both sending data to C, as fast as they can, via switches S1 and S2 (which are connected via a high-speed link). And say the sum of these two flows is more than the capacity of the link to C.

S2 is receiving packets destined for C faster than it can forward them, but sending an Ethernet pause frame from S2 to S1 is not a very productive way to alleviate the situation, because it also disrupts any traffic that would be bound for D. It just moves the bottleneck elsewhere and causes collateral damage.

View on HN

Maybe, but doesn’t RoCE / RDMA handle this at a higher level as well? It’s why PFC and ECN are required for these deployments, which effectively ensure everyone behaves nicely.

I agree that Ethernet flow control is insufficient, but given that NVidia in an act of brilliant foresight acquired Mellanox a decade ago, I’m fairly certain that this is how all these AI clusters are actually deployed, not using TCP, and maybe not even using Ethernet but infiniband instead.

if you spend the zillion dollars on switches and racks and power and cables and redundant power in your datacenter for infiniband(the only vendor: Nvidia), you get IB.

if not, you get RoCE over ethernet.

... which brings the question, "how is X solved?".

For IB the answer is generally a centralized controller, a lossless flow control algorithm and the credit system with subnet coordination. Ie. the packets that would be dropped due to buffer problems never get sent because the originating node knows it's out of credits. This means packet drops should be nonexistent and so the protocols make the assumption only one packet in a zillion ever gets dropped. On the down side: the tiniest of problems on IB networks have incredible impact (but it's realistic to just not have those)

IB doesn't interface with others. Yes, there's an ethernet bridge but it's not going to work nearly as well once you insert that bit of kit.

In ethernet packets are dropped when congestion is experienced, and you configure through priority queues whose packets get dropped first (you configure priority), and "stuff will sort itself out" like on the internet. RoCE tries to construct IB semantics from this, but there's limits. On the plus side: this is how the internet works and so the whole actually works somewhat reasonably over internet or other long-distance links (and tunnels into Neoclouds and/or Clouds). On the downside it's never going to match IB in speed (it's still going to get to 90% or so link speed, 98% if you just naively check the link utilization and ignore the effect retransmissions will have)

> if you spend the zillion dollars on switches and racks and power and cables and redundant power in your datacenter for infiniband(the only vendor: Nvidia), you get IB.

Worth noting that Nvidia/Mellanox is currently the only vendor for IB, but that has not always been the case. I had Intel IB switches for the backend network of my Isilon at my last job (we starting using them when they were 'just' Isilon, then EMC Isilon, and now Dell-EMC Isilon); only post-Dell did they start going Ethernet-only, and now with AI/ML they're re-offering IB as an option (LOL).

So Nvidia is kind of being 'rewarded' for sticking with a technology that everyone gave up on (I guess HPC wasn't big enough to bother with).

Besides IB, the other options being used in Top 500 are Slingshot (Eth-base?) and Omni-path (Category: Interconnect Family):

* https://www.top500.org/statistics/list/

I hope whoever led the Mellanox acquisition at Nvidia is treated internally like a prophet.

Reply on news.ycombinator.com