Hacker News Reader

Comment by bhouston | Hacker News Reader

I love the PCIe standard is 3 generations ahead of what is actually released. Gen5 is the live version, but the team behind it is so well organized that they have a roadmap of 3 additional versions now. Love it.

View on HN

"3 generations" seems like a bit of a stretch. Millions of Blackwell systems use PCIe 6.x today, PCIe 7.x was finalized last month, and this is an announcement work on PCIe 8.0 has started for release in 3 years. I.e. it has only been one month of being one generation behind the latest PCIe revision.

It'll be interesting if consumer devices bother trying to stay with the latest at all anymore. It's already extremely difficult to justify the cost of implementing PCIe 5.0 when it makes almost no difference for consumer use cases. The best consumer use case so far is enthusiasts who want really fast NVMe SSDs in x4 lanes, but 5.0 already gives >10 GB/s for a single drive, even with the limited lane count. It makes very little difference for x16 GPUs, even with the 5090. Things always creep up over time, but the rate at which the consumer space creeps is just so vastly different from what the DC space has been seeing that it seems unreasonable to expect the two to be lockstep anymore.

either I'm misinformed in which case is like to see proof, or you are, but GB200 is not Gen6, just Gen5 and this will only be rectified with GB300 which, while exists in the wild, is not in the millions.

so indeed the parent commenter would be correct that everything is Gen5 right now.

I'm going off of https://developer.nvidia.com/blog/nvidia-gb200-nvl72-deliver... but I have not actually deployed a GB200 system yet myself. Consumer Blackwell is definitely 5.0 though.

that article is from last March. the plan was indeed for the GB200 board to support gen6 but it didn't happen and GB300 will rectify it.

at least that's my best educated guess, looking at supermicro's public spec sheet and that it's shipping with CX7 which is Gen5 and not with CX8.

B200 supports Gen6 there is just nothing that would let it run at Gen6.

At least the GB200s OpenAI purchased have CX-8s, but it's very possible (perhaps even very likely) most of the millions of GB200s deployed PCIe 5.0 using CX-7s given the SuperMicro spec sheet.

Edit: found this SemiAnalysis post saying as much https://x.com/SemiAnalysis_/status/1947768988467138645/photo...

thanks! very interesting that they got a custom SKU.

Datacenter Blackwell is Gen6, which is critical when pairing it with CX8 (2x400G) as otherwise you'd be stranding NIC BW.

please link me a B200 (not a B300) system with CX8 that uses gen6 x16 instead of gen5 x32.

to the best of my knowledge this does not exist, but I'd be happy to stand corrected.

(the official NVIDIA DGX B200 is gen5).

CPUs themselves only have so many PCI-E lanes though right? Wouldnt it make sense (even for consumers) to have peripherals using less lanes (but more speed per lane) for a multi GPU system or something that uses a lot of drives?

More lanes = more cost

Faster lanes = more cost

More faster lanes = lots more cost

The chipset also strikes some of the balance for consumers though. It has a narrow high speed connection to the CPU but enables many lower speed devices to share that bandwidth. That way you can have your spare NVMe drive, SATA controller, wired and wireless NICs, sound hardware, most of your USB ports, your capture card, and some other random things connected over a single x4 to x8 sized channel. This leaves the high cost lanes for just the devices that actually use them (GPU and primary, possibly secondary, storage drive). I've got one consumer type Motherboard with 14 NVMe drives connected, for example, just not at full native speed directly to the CPU.

You're just SoL if you want to connect a bunch of really high bandwidth devices simultaneously (100 Gbps+ NICs, multiple GPUs at full connection speed, a dozen NVMe drives at native speed, or similar) because then you'll be paying for a workstation/server class platform which did make the "more faster lanes" tradeoff (plus some market segment gouging, of course).

One issue is that, at least on cheaper mobos, these don't work as a "total bandwidth budget" situation. And, especially with newer generation PCIe, it can be a bit frustrating.

Many mobos will operate the available slots such that the total number of active lanes is split between them. But if you use older-generation cards, you'll only get a fraction of the available bandwidth because you're only using a fraction of their lanes, although the physical lanes are physically present.

What I'm thinking about is something like, say, a pair of Gen3 NVMe drives that are good enough for mass storage (running in RAID-1 for good measure) and some cheap used 10 Gb NIC, which will probably be 8x Gen2, all running on a Gen4+ capable mobo.

And, while for a general-purpose setup I can live with splitting available BW between the NIC and the GPU (I most likely don't care about my download going super fast while I game), the downloads will generally go to the storage, so they must be fast at the same time.

Those MBs are cheaper precisely because supporting this kind of bandwidth breakout adds cost (-> a fancier PCIe switch in a higher end chipset/southbridge). If you add the support to do this to them you end up with the more expensive motherboard. Some of the highest end motherboards actually have 2 chipsets/PCIe switches, more cost but a more bandwidth sharing for the same number of lanes coming from the CPU.

You can also buy external PCIe switches (just make sure you're not accidentally buying a PCIe bifurcation device). Most of the time it's cheaper to just buy the higher end motherboard though, e.g. I don't want to know what price "Request a quote" for this PCIe switch which can do x8 4.0 upstream and then quad 4x 3.0 downstream https://www.amfeltec.com/pci-express-gen-4-carrier-board-for... is. I do have a few 3.0 era cards which were more reasonably priced though https://www.aliexpress.us/item/3256801702762036.html?gateway... and they've worked well for me.

How high-end are we talking about? Do you know off-hand of any models supporting this?

I haven't seen such features on boards under 200 EUR, from Asus, Asrock and Gigabyte.

The thing is, if I have to splurge for some 400 EUR "gaming" model, I might as well move to a "workstation" CPU supporting more lanes out of the box, and the mobo will be priced roughly the same.

> "3 generations" seems like a bit of a stretch. Millions of Blackwell systems use PCIe 6.x today

True. And yet, if you buy an RTX 5090 today, costing $2400 and released in January this year, it's PCIe 5.0 x16

Does the second half not already talk to this? Adding on to it further, GN found a few % performance difference on a 5090 going from PCIe 3.0 to 5.0 https://gamersnexus.net/gpus/nvidia-rtx-5090-pcie-50-vs-40-v... meanwhile many DC use cases find PCIe 5.0 unusably slow, driving alternative (non-PCIe) interconnects.

It takes a long time to get form standard to silicon, so I bet there are design teams working on pcie7 right now, which won't see products for 2 or more years

Is there an advantage of getting so far ahead of implementations? It seems like it would be more difficult to incorporate lessons.

When AMD introduces a new Desktop CPU series IIRC they claim the next generation design is (almost) finished (including layout?) and they start with the next-next-gen design. And I'm also asking the same question. But more than a half a year before the CPU becomes available to the public it is already being tested by partners (mainboard manufacturers and ?).

Gen6 is in use look at Nvidia ConnectX-8

What hosts support Gen6? AFAIK, Gen5 is the most recent standard that's actually deployed. Eg, what can you plug a CX8 into that will link up at Gen6?

Custom Nvidia network cards I guess.

Blackwell DC (B200/B300)

while the B200 chip itself could do PCIe6 as it was planned for GB200, there is no system around it with Gen6. the official DGX B200 is just PCIe5.

GB300 is indeed Gen6.

into a Gen6 Bianca board (baseboard for GB300). that's it today.

how you use those today is twofold:

- Gen5 x32 via two x16 slots (this is how most people use them)

- use actually the CX8 as your PCIe switch directly to your GPUs: https://www.servethehome.com/this-is-the-nvidia-mgx-pcie-swi...

Nvidia only for now. Precisely because they got fed up with Intel and AMD ( And Amphere as well ) for extremely slow PCIe adoption.

I believe only next Gen Intel and AMD Zen6 will get PCIe 6.0.

I am hoping Nvidia officially move into Server CPU market not only for their CPU but for wider Web Hosting as well. More competition for Server Hardware.

This actually makes sense from a spec perspective if you want to give enough to allow hardware to catch up with the specs and to support true interop.

Contrast this with the wild west that is "Ethernet" where it's extremely common for speeds to track well ahead of specs and where interop is, at best, "exciting."

So we can skip 6 and 7 and go directly to 8, right?

Reply on news.ycombinator.com