Hacker News Reader

Comment by riku_iki | Hacker News Reader

What router/switch one would use for such speed?

Linked article says they used 68 machines with 2 x 100GbE Mellanox ConnectX-6 cards. So any 100G pizza box switches should work.

Note that 36 port 56G switches are dirt cheap on eBay and 4tbps is good enough for most homelab use cases

> So any 100G pizza box switches should work.

but will it be able to handle combined TB/s traffic?

any switch which can't handle full load on all ports isn't worthy of the name 'switch', it's more like 'toy network appliance'

I will forever be scarred by the "Gigabit" switches of old that were 2 gigabit ports and 22 100mb ports. Coworker bought it missing the nuance.

Still happens, gotta see if the top speed mentioned is an uplink or normal ports.

Yes. Most network switches can handle all ports at 100% utilization in both directions simultaneously.

Take for example the Mellanox SX6790 available for less than $100 on eBay. It has 36 56gbps ports. 36 * 2 * 56 = 4032gbps and it is stated to have a switching capacity of 4.032Tbps.

Edit: I guess you are asking how one would possibly sip 1TiB/s of data into a given client. You would need multiple clients spread across several switches to generate such load. Or maybe some freaky link aggregation. 10x 800gbps links for your client, plus at least 10x 800gbps links out to the servers.

Even the bargain Mikrotik can do 1.2Tbps https://mikrotik.com/product/crs518_16xs_2xq

For those curious, a "bargain" on a 100gbps switch means about $1350

On a cluster with more than $1M of NVMe disks, that does actually seem like a bargain.

(Note that the linked MikroTik switch only has 100gbe on a few ports, and wouldn't really classify as a full 100gbe switch to most people)

Sure- I don't mean to imply that it isn't. I can absolutely see how that's inexpensive for 100gbe equipment.

That was more for the benefit of others like myself, who were wondering if "bargain" was comparative, or inexpensive enough that it might be worth buying one next time they upgraded switches. For me personally it's still an order of magnitude away from that.

https://mikrotik.com/product/crs305_1g_4s_in is the sweet spot right now for home users. Four 10g ports and a 1g, you can use the 1g for “uplink” to the internet and one of the 10g for your “big old Nortel gigabit switch with 10g uplink” and one for your Mac and two for your NAS and VM server. ;)

Direct cables are moderately cheap, and modules for 10g Ethernet aren’t insanely expensive.

there's usually some used dx010 (32x100gbe) on ebay for less than $500

the cheapest new 100gbe switch I know of is the mikrotik CRS504-4XQ-IN (4x100gbe, around $650)

TB != Tb..

Given their configuration of just 4U spread across 17 racks, there's likely a bunch of compute in the rest of the rack, and 1-2 top of rack switches like this:

https://www.qct.io/product/index/Switch/Ethernet-Switch/T700...

And then you connect the TOR switches to higher level switches in something like a Clos distribution to get the desired bandwidth between any two nodes:

https://www.techtarget.com/searchnetworking/definition/Clos-...

800Gbps via OSFP and QSFP-DD are already a thing. Multiple vendors have NICs and switches for that.

can you show me a 800G NIC?

the switch is fine, I'm buying 64x800G switches, but NIC wise I'm limited to 400Gbit.

fair enough, it seems I was mistaken about the NIC. I guess that has to wait for PCIe 6 and should arrive soon-ish.

16x PCIe 4.0 is 32GB/s 16x PCIe 5.0 should be 64 GB/s, how is any computer using 100 GB/s ?

I was talking about Gigabit/s, not Gigabyte/s.

The article however actually talks about Terabyte/s scale, albeit not over a single node.

800 gigabits is 100 gigabytes which is still more than PCIe 5.0 16x 64 gigabyte per second bandwidth.

You said there were 800 gigabit network cards, I'm wondering how that much bandwidth makes it to the card in the first place.

The article however actually talks about Terabyte/s scale, albeit not over a single node.

This does not have anything to do with what you originally said, you were talking about 800gb single ports.

Yes, apparently I was mistaken about the NICs. They don't seem to be available yet.

But it's not a PCIe limitation. There are PCIe devices out there which use 32 lanes, so you could achieve the bandwidth even on PCIe5.

https://www.servethehome.com/ocp-nic-3-0-form-factors-quick-...

I'm not aware of any 800G cards, but FYI a single Mellanox card can use two PCIe x16 slots to avoid NUMA issues on dual-socket servers: https://www.nvidia.com/en-us/networking/ethernet/socket-dire...

So the software infra for using multiple slots already exists and doesn't require any special config. Oh and some cards can use PCIe slots across multiple hosts. No idea why you'd want to do that, but you can.

Reply on news.ycombinator.com