Hacker News Reader

No, as in 4 GPUS and/or one or more PCIE NVMe storage devices or PCI based comms cards such as infiniband cards.

This. I've been trying to put together a small home lab xeon e3 system that can theoretically support nvme+gpu+infiniband. It's tough.

Typically you get something like 3 full length PCIe slotson a xeon e3 board (c236 chipset). BUT they can't all run full speed at 16x (16/16/16). You have to then choose between 16/8/8 and 16/16/0 due to the lack of PCI lanes.

Can you explain why you need 16/16/16? Wikipedia tells me that one x16 PCIe 4.0 connection has a capacity of 32GB/sec (x8 is 16GB/sec). A quick search shows NVMe drives with a sustained read or write speed of ~2-4 GB/sec. Equally for the GPU - you need to move a lot of data in/out of the GPU to saturate a x8 connection.

It's rare to run that sustained, usually it's bursty so the extra bandwidth is very much needed.

Reply on news.ycombinator.com