The Path Is Set for PCI-Express 7.0 in 2025

PCI-E speed is starting to push against the noise of the universe.

At these kinds of bandwidths, it is extremely noisy.

Remember socket AM4, which added support for PCIe 4.0 with chipset X570? Originally, the idea was that non-X570 boards would also have PCIe 4.0, as it doesn't actually depend on the chipset - just the CPU. Turns out most boards simply weren't capable of handling the additional bandwidth, so it was eventually restricted to boards which were designed for it. And that's at one-eighth the bandwidth of PCIe 7.0!

If it were truly a bus, these signaling rates would be totally impossible. Too many reflections.

In practice it's a bunch of point-to-point links, but even then the path across a PCB is pretty brutal in terms of loss. Cables are (surprisingly to me, initially!) better.

Overall though, sending some number of additional bits for error correction is often cheaper than driving the line harder to reduce the bit error rate.

Put differently: we could manage lower noise, but it would either result in a lower bitrate or (much) higher power. Instead, pick a target goodput and design for power and error correction from both sides.

It says "GHz" so yes it's extremely noisy.

At this point the bits are millimeters long.

Noise is a big one. The more levels you have, the more likely it is noise is going to turn a 0 to a 1 or a 1 to a 2, for example. It also requires more complicated switching transistors, because you can't rely on things being simply on or off, but rather have to generate a number of intermediary voltages.

The more complicated the modulation is, the more processing power is required. DSL modems use very complex modulation to push their data across the wire and burn a few Watts doing so (there's also a fairly powerful line driver, but that's not the main power consumer, the DSP/modem itself is).

Of course, SNR is another big one. You have to actually be able to distinguish the voltage levels within a symbol period from each other.

Another thing: PCIe is meant to be low-latency, which just rules out a lot of the modulation and error-correction techniques that are traditionally used to enhance channel utilization, like having multiple levels of FEC and then interleaving FEC symbols temporally. That's done basically always where latency allows, e.g. in storage media, digital TV and so on.

He already said it. Noise.

With 2 levels you have low and high. I have no idea if PCIe is 1.8V but let’s be really gracious here and pretend it’s 5V.

Ok; so at 5V and 4 levels you have Sig Al’s between 0V, 1.25V, 2.50V, and 3.75V. And are encoding 2 bits per clock.

At 64 levels you have a signal every 0.078125V. It’s going to be really hard to capture that signal at a moment in time and at these speeds be certain if it’s definitely A or B.

At some point you get into modems (modulation, demodulation) but the add much more cost, latency, and complexity.

So before getting into radio and analog tech, here is a simple way to double the bandwidth for little cost. A driver only needs 0,1,2,3 levels and a receiver can make fairly confident guesses quickly.

I’m pretty sure PCIe isn’t 5V. Probably more like 1.8V, so already starting with a much smaller range.

Differentiating between black and white is easy. Distinguishing between fifty shades of grey is difficult.

The other answers miss the point.

A flip flop can be calibrated to the difference potentials of thr input. You have zero and one as a result, no processing required. Single bit, single voltage. Simple.

Think about what it means to map a single voltage input to multiple bits. What device would do it? A complex ADC? Think of the overhead that would introduce.

It's also similar to how gigabit+ ethernet, ADSL, and wireless comms work (wireless and ADSL go much further, they also vary the phase of the signal and the constellations can have >1024 points in extreme circumstances).

The generalization of this idea:

https://en.wikipedia.org/wiki/Quadrature_amplitude_modulatio...

Yes, but the average PC user doesn’t do that. It’ll take some time until we get PCI-e 7 NVMe sticks.

At least 2.5 GBe is rapidly catching on in home networking, with more and more motherboards getting it by standard along with Wifi 6e routers that can finally exceed 1 Gbit real-world

Wifi routers are plagued by TERRIBLE 1% latency figures.

I'm far more concerned with wifi not dropping out than mean performance.

> I thought it is worth mentioning with a dedicated Storage API and SSD on the PS5, 90+% of loading time are spent on CPU vs very little on I/O.

With faster buses and SSDs, this is probably going to get worse.

I wonder if we'll see a CPU breakthrough within our lifetimes (let's say, before 2050).

I kind of doubt we'll actually see cost savings. PCI-e 4.0 SSDs tend not to be 2 lanes vs 4 lane PCI-e 3.0 SSDs. Intel went from pci-e 4.0x20, dmi 3.0 x8 on LGA 1200 to pci-e 5.0x16, pci-e 4.0x4, dmi 4.0 x8; so same number of lanes, but most of them twice the speed. AMD has 24 lanes of pci-e in AM4 (at 3.0 or 4.0), and AM5 looks to also have 24 lanes of pci-e, but at 5.0.

So we're going to get the same number of lanes, they'll be twice as fast (more or less), and we'll have to pay more for them.

I do see that some dual 10G ethernet cards are pci-e 2.0 x8, and some are pci-e 3.0 x4, but I can't find solid pricing for like for like to see if that does actually reduce costs.

I'm no expert but I bet some users would enjoy RAM over a switchable PCIe fabric. They'd take a latency hit but would have the potential for massive amounts of memory.

Many things. Capacity, Bandwidth, Serviceability [1]. It also paves the road towards disaggregated memory [2].

edit: The official Compute Express Link (CXL) website has an article about the benefits of this technology [3].

[1] - https://www.nextplatform.com/2021/09/07/the-cxl-roadmap-open...

[2] - https://pmem.io/blog/2022/01/disaggregated-memory-in-pursuit...

[3] - https://www.computeexpresslink.org/post/the-benefits-of-seri...

Amd's A300 and X300 chipsets for AM4 are just a bit of rom. Modern CPUs don't really need a chipset; AM4 chipsets connect to the CPU through PCIe, anyway.

We'll be right back in the S-100 bus days

AMD EPYC is already that, a SoC - there is no EPYC chipset. They all have 8x 16-lane multi-function interfaces, which can be configured to work as PCIe, xGMI (inter-socket communucation), or SATA, as well as built-in USB, eMMC, LPC, UART, I2C, SMBus, GPIO.

Memory latency is only bad when you have a cache miss. That’s why mainframes have gigabytes of cache between their processors and memory.

You can mitigate that with some in-package HBM. IIRC, Xeon Phis came with up to 16GB of it - you could in theory have a workstation without any memory sticks.

I would hope and expect that CXL gets latency low enough to make it viable. These days you're mostly waiting for the core of the ram chips to respond as everything else races around it.

For Optane, I'd say it's doing so badly because it's barely cheaper than RAM, which means it's only useful in niche situations.

Is it possible one manufacturer "takes the leap" and provides ram modules that are faster than competition. Therefore driving the transition purely on competition alone?

There's a significant momentum behind CXL [1]. Samsung already announced a CXL DRAM module [2]. From where I sit, it's all but inevitable that this technology will get broader adoption, at least with large CSPs. Its benefits are just too good to pass up. The extra latency is estimated to be no worse than a NUMA hop [3]. Many companies are already figuring out how to best deal with the more complex memory topologies that CXL will enable [4, 5].

But I'm biased (I work for Intel), so take what I say with a grain of salt.

[1] - https://www.computeexpresslink.org/members

[2] - https://news.samsung.com/global/samsung-electronics-introduc...

[3] - https://www.lenovoxperience.com/newsDetail/283yi044hzgcdv7sn...

[4] - https://arxiv.org/pdf/2203.00241.pdf

[5] - https://arxiv.org/pdf/2206.02878.pdf

> They're too cheap to use differential signaling even on GDDR memory.

Makes it quite impressive that they're able to push 20+ Gigabit/s/pin even in dual-sided / T-layouts (though that also uses PAM4). Of course, comparing historical GPU layouts it's pretty obvious how the RAM chips are creeping ever more closely to the GPU die - weird how that is.

The DRAM signaling is very strange, I worked briefly on a hardware DDR controller and the timing reference was over pins called dqs. These pins sent a signal to represent the bit transition so they could be safely captured by the receiver. All good, except the pins were bidirectional so when transmitting you sent a reference on the same pins, so lost the reference from the other side. The issue then was when you just switch from trasmitting to receiving, you have to quickly re-lock to the sending reference, which is a pain in the ass, and really complicates things. DDR phys could be simplified if dqs was unique per TX and RX directions, but nope, penny pinching on the signal count.

There is a long tradition of serial memory interfaces. RDRAM/XDR DRAM, FB-DIMM/AMB, Intel SMB, OMI DIMM/DDIMM, etc.

They seem to have most success in high end server CPUs where memory capacity and throughput are important. They have cost, power, latency disadvantages, so their forays into low end servers, PCs, etc has been more sporadic.

Forgive me if this is an ignorant question, but couldn't you use PCIe RAM like cache in the same way a CPU's cache levels work? It's not as fast, but could feed the regular RAM faster than from another source.