Hacker News Reader

It's a shame you can't get a cable to link computers via PCIe (supports cables over 3m[0]). Multiple x16 connections that would lead to some decent bandwidth while being low-latency, without the network overhead or shelling out for high-end switches.

You can skip halfway. [0]https://www.youtube.com/watch?v=q5xvwPa3r7M

View on HN

Using dual port adapters, you can do a three-node Infiniband ring with no routers involved and get full speed. Dual port FDR (40 Gbit/s) cards are about $150 on Ebay nowadays. If you've bought the hardware to warrant needing such a setup (i.e. 3 dual-socket Xeon servers), that's a very negligible cost.

PCIe fabrics exist but they're not as great as you might expect. Because of the slow 8 Gbps lane speed the cables are enormous and the switches support very few ports. And the DMA engine may not give you the performance you expect if it doesn't have NIC-like features.

https://www.broadcom.com/applications/datacenter-networking/...

Reply on news.ycombinator.com