Hacker News Reader

Comment by userbinator | Hacker News Reader

less than 60ns loopback latency

There are some Ethernet switches with 4ns latency, and those do more than just sending and receiving, so there's clearly still an order of magnitude of improvement still available. 4ns is basically ~40 cycles of the bit clock for 10G Ethernet.

View on HN

It's a bit more complicated than that. Loopback implies that the data went off the wire and was sent back. That puts some serious constraints on the process that deals with that data, how it is stored and how it is fed back into the system. You can't just compare that with naked pass-through latency (which is what a switch would do), because the signal is already on the way out before it has even fully been received.

Essentially a few ns after the header is received it will already be passed on to the output port after some signal conditioning, which happens almost without latency. A typical 'high speed' switch will have 800 ns round trip, so that 60 ns quoted here is actually quite impressive.

There are L1 crosspoint "switches" in the 2-5ns range depending on port density and similar modes in some Ethernet silicon. These are not Ethernet switches in any normal sense though. They only replicate the signal 1:1 or 1:n and do not dynamically switch the destination based on anything in the packet. The fastest L3 Ethernet switches on the market are ~90ns.

Are L1 switches like hubs or repeaters then? Sorry for my confusion.

They're more like digital patch panels. You can connect any pair of ports together in both directions (TX/RX), just like moving a patch cable around. You can also connect an incoming RX lane to multiple TX lanes (think optical splitter but electrical). You cannot merge signals together. Some products have FPGAs that add L2+ or multiplexing functionality with packet buffers, but that's not part of the 2-5ns path.

So, if I know the destination addresses on each port in advance I can program it to behave like an L2 switch (with proper multicast even), and I can use code to reprogram it whenever the traffic pattern changes?

The L1 path can't do any address matching and you can't overlap sources - like Ghostbusters, no crossing the streams. You can source port 1 to ports 2-4 (multicast, but static and unidirectional) and you can change a bidir connection between pairs of ports. The signal gets replicated regardless of what's in the frame. It wouldn't be practical to make it behave quite like an L2 switch.

Apologies, my post was not precise enough, but I think I got it.

> It wouldn't be practical to make it behave quite like an L2 switch.

What are typical configuration? If you’re allowed to talk about it.

No they don’t. At least not on the critical path. L1 switch’s at 4ns simply replicate the incoming electrical signal. That’s all they do.

They also do some packet accounting and cute features off the critical path. None of that at 4ns.

This guy knows a thing or two about these devices ;)

Yes he very much does :)

In fairness, assuming 4/8 ports in, 4 ports out, operating at some ungodly GHz using a custom GaAs or SiGe chip, and working in gearboxed scrambled space with very clever input mac prefix mapping and output scrambled/gearboxed precomputation, one _could_ do around 8ns optimistically from the start of the first 64b/66b block (the 0x55... preamble) for 10GB. There's some stuff about preamble shrink that makes it wonky, but that is doable. And interestingly enough, for 25G-R, one can comfortably do this in about 4ns. I am not in fact aware of such a beastie existing for 25G, but I have seen 1 or 2 for 10G though.

Surprisingly, if ChatGPT is prompted _juust_ right, it will even give you a good way to do this.

The other funny bit is that one-way PCIe latency is 250ns-ish (don't quote me on the exact numbers), which imposes a hard 1us constraint on latency between two hosts.

You can go quicker with CXL, but not by much.

There are always DPUs like NextSilicon, NVidia's Blue*, etc - at under 100 ns of SoF to fast compute though.

> There are some Ethernet switches with 4ns latency […]

E.g.,

* https://www.arista.com/en/products/7130-connect

* https://www.arista.com/en/products/7130-series

Cut Through switching needs what, 14 bytes? Thats ~12ns on 10G? How would you do it in 40 cycles?

Not my are of expertise, but afaik the neat part is that you don't do it in cycles. The "program" lives as the transistor layout, not as instructions

The cycles in question are not those of any processor, but those of the signal coming in the wire. No matter how it's implemented, a switch cannot react to bits that are still in the wire and haven't made it to the switch yet.

why do you assume the full ethernet header?

IIRC. cut-through only needs the first 6 bytes. Since it only needs the destination address for the port lookup.

Potentially the first bit, on broadcast.

Because you might want VLAN. Plus you cant just start blasting reply without a preamble, so its still 14 bytes after receiving just the dest MAC. Then you get potential IFG, FEC, scrambling, it all adds up, no way any switch can do 4ns without heavy lawyer talk in the small print.

For switching, this is just not true. There's in/out solutions that are at this kind of scale, but they are by definition not implementing any switching logic.

Reply on news.ycombinator.com