Comment by justinclift | Hacker News Reader

Comment by justinclift | Hacker News Reader

Parent Full thread

> I don't agree [...] absolute highest performance is found in Infiniband, not only with bandwidth but latency.

What's the alternative(s) you're thinking of?

There are some specialist things inside supercomputers (in the cases that Infiniband isn't used), but there doesn't seem to be anything else reasonably available commercially.

Omni-path (a descendant of Infiniband) doesn't seem to have worked out either.

---

Infiniband also has advantages over Ethernet in other things, like bundling links together.

With (standardised) ethernet, adding more cables/links between two hosts gives more overall bandwidth in total. But the additional links can't be used to increase the bandwidth of a single stream.

With Infiniband, when you add links they just combine together into a single bigger pipe, which can transfer more traffic in a single stream. Less need to jump up to higher tier adapters and switches with IB for that, you can just run more links between the hosts.

I re-worded what I wrote for clarity, it's an obscure sort of sentence construction.

"I don't agree with Jim Keller in that absolute highest performance is found in Infiniband, not only with bandwidth but latency. "

I don't agree with Jim Keller who says that Ethernet would be as good or better, in that when you use Infiniband, you get the absolute highest performance from IB, since IB does best on bandwidth and latency considerations.

No worries, my misunderstanding. :)

You can definitely bond multiple Ethernet links together ("IEEE 802.3ad link aggregation").

Sure, but ethernet doesn't let you combine the entire bandwidth of those links into a bigger fat pipe for a single stream. It more designed for higher bandwidth going to multiple destinations.

A super common example of this is when consumers want to upgrade their existing home infrastructure to use a new NAS, and they start hitting the limits of 1GbE. Their first question is often along the lines of "can't I just add more links together to give more bandwidth?".

Then people need to explain "No, ethernet doesn't work that way. You need to move up to 2.5/5/10GbE". (etc)

Infiniband doesn't have that problem.

It, not Ethernet that doesn't support the bandwidth aggregation you want, but the protocol over it. Multi Stream TCP seems to be one solution to this problem

This used to be done, using what was called round-robin mode in bonded links. I think you are right that this mode never found its way into 802.11ad. Instead, parallel stacks of switches would be deployed, so that hosts were configured to bond links to N switches but the switches were not aware of the bonding. It was a commonly used Linux option to modestly scale up with low cost hardware.

A single stream would have its frames striped over all the links in the bond, due to the configuration of the sending host. The flows would remain parallel and arrive over N switches to N NIC ports on the destination host. Care had to be taken with topology to keep parallel paths from end to end such as in one datacenter. Generally, flows could fan-in somewhere on the path and then go over a higher speed uplink, but would never fan-out again to parallel links.

The drawback is more frame (packet) reordering, so it is even more sensitive to buffering and congestion control effects. If there were switch failures, the flows could become unbalanced if the topology offered redundant cross-links, since this would cause some fan-in on the surviving switches. Otherwise, completely independent switch stacks could be used, so a failure means traffic is dropped. Then, the host needs to do some kind of remote peer connectivity check to decide whether each link is healthy or not, and drop unhealthy links from the bond.

Ironically, this could be done in consumer-grade LANs, since it does not depend on bonding features of the switches. However, careful setup is needed with consideration of the topology and switch behavior, to avoid pathology.

Reply on news.ycombinator.com