So -- how do you prove that your infrastructure can blow the socks off everything without building two of them and testing?
So -- how do you prove that your infrastructure can blow the socks off everything without building two of them and testing?
Any communication protocol has functional layers to them, and at the very bottom is what's called the Phy layer (or physical layer.) While pretty much every layer can be implemented in digital logic, the physical layer has a lot of analog circuitry. (It's programmable analog circuitry, so it's technically mixed signal.) Most wired communication protocols that you are familiar with (including SAS/SATA, USB, HDMI/DP, PCIe/NVMe, and Ethernet) all use what's known as a "SerDes Phy" which is short for Serializer Deserializer. It takes parallel data in, and serializes it to transmit data, and does the reverse when receiving data. In the example of PCIe Gen 4, it takes 32 bits of data in at 500 Mb/s (per bit) and serializes it to 16 Gb/s.
Because the SerDes Phy is such a common part of all communication protocols, it's often reused in other designs. For example - a GPU may use the same SerDes Phy design for its PCIe lanes as it does for its HDMI ports (at least on the transmit side.) They are however programmed differently.
Because these designs see so much reuse in so many different chips, there are companies that specialize in designing SerDes Phys, and their development is usually ahead of protocol development. Though in today's day and age, it's super complicated, and consists of way more than shift registers and a PLL. But there are SerDes designs that exist in products that are capable of 112Gb/s (not sure if 224Gb/s exists outside of test chips yet.)
You can buy modern FPGAs that have PCIe Gen 5, but with a SerDes Phy capable of going way faster. And if you want, you could bypass that hard block and talk straight to the SerDes, and program your own PCIe 6.0 controller.
But to answer your question on how they prototype it and test it, they run computer simulations that are exceedingly complicated and slow, and once a test chip comes back from the fab, eye diagrams are usually collected using what's known as a repeating signal scope. Since it's insanely difficult to sample a 112Gb/s signal at any reasonable resolution, it's sampled statistically (in time) over billions of samples, and plotted by knowing the frequency of the target signal. An eye diagram is what comes out of a traditional NRZ (non return to zero) where each cycle represents either a 1 or a 0. In an eye diagram, you see all permutations of bit transitions overlayed on each other, since you're sampling billions of cycles statistically. This is important because you can see other anomalies of your signal like jitter and rise/fall times.
There's a lot more to it, and this comment is already getting to be a wall of text but definitely feel free to ask more questions!
Could switching to PCIe instead of Ethernet for the digital signaling have better performance? I don’t know but I doubt it. Ethernet is hitting 100gbps and higher and I suspect the challenge is designing cabling that can hit just as much as the analog design piece. Not to mention that USB3.2 and Ethernet have very similar speeds, just Ethernet can manage it over much longer distances at significantly lower cost. I’m skeptical PCIe has some kind of magic bullet here. These protocols are optimized for totally different use cases.
I think the reason why it costs a lot is that packets are still tiny, and ports are many. A quality 16 port, 10 Gb switch is expected to have no internal bottleneck, able to handle every port being used to full capacity at once. And that works out to 320 Gbps bandwidth in the whole switch.
Then there's that thanks to the Internet, we're still stuck with 1500 byte packets, which means best case we're dealing with 812,744 pps, and worst case, we're dealing with 14,880,952 pps. And that's on one port.
With the later number, your time budget per packet is 67ns. And if you start looking, context switches, cache misses, locks, etc all have times also measured in nanoseconds.
It's getting very, very tight and requires extremely careful programming to actually make this work. And of course in modern times, we have firewalls, NAT, statistics, routing, etc all of which cost precious time out of this budget, and which are all user-configurable, so it's something that's difficult to just shove into a purpose-built chip to take that load off the CPU.
PCIe has the advantage of that it doesn't have to deal with half the crazy stuff networking involves.
> 16 port, 10 Gb ... 320 Gbps bandwidth in the whole switch
That's a nice switch. Expensive, but nice. Let's do the math on a modern low-end consumer CPU:
PCIe: 20x 32Gb/s x2 = 1280Gbps
Chipset: 8x 16Gb/s x2 = 256Gbps
Display: 4x 4x 12Gb/s = 192Gbps
...and if we compared the switch to a CPU in its own price bracket, this would go from silly to ludicrous.
Look, I'm not asking for a battery of 400GbE ports on every laptop, but USB can deliver 10Gb/s across a few meters for a few dollars while Ethernet wants a few hundred dollars to ship the same bits over the same distance, and that's unfortunate.
People keep trying to make excuses for Ethernet, but I don't think it deserves their efforts. I think something's broken in the consumer networking industry -- it feels like they gave up on consumers entirely after 1GbE, and I think we could all benefit if someone breathed some life back into that corner of the market.
How is it even possible to have correctly functioning circuits at such high speeds? I'm guessing at the very bottom of the Phy layer there several transistors switching at incredibly fast speeds. I understand that these transistors have several parameters such as their delay, rise, and fall time. Things most certainly don't happen instantaneously, a few logic ICs I played with also have a propagation delay time specified in their data sheets lasting a few nanoseconds.
Take the deserializer for an example, happening at 16Gb/s, which if I understand correctly equates to 1 bit every 62.5ps (0.0625ns). How does any circuit affected by the parameters I just talked about handle signals at such high speeds?
(Please go easy on me, I know some of my assumptions are probably wrong or oversimplified but I would love to learn more)
One more thing: CMOS historically wasn't the fastest logic family. Bipolar ECL was faster, but it burned a ton of idle current, so it ran into power limitation long before CMOS.
In either case, thermal limits prevent you from using these speeds across the whole chip, but just because your thermal budget can't afford 10,000,000,000 speed demons doesn't mean it can't afford 1000 of them in a few SerDes :)
But yeah, decades of Moore's law have not gone to waste, and those discrete logic chips are frozen in time. It's a neat reminder of how far we have come!
I work on the Hardware/Software interface quite a bit (more in the past than now) and this is always fascinating, I'm on the Software side, thanks for the details!
The encoding takes 8 bits and turns them into 10 for DC balance and to help the clock recovery system. You are basically losing 20% of your bandwidth to encoding overhead.
PCIE Gen 3 and above have switched to 64b/66b encoding for less overhead.
Generally, if you are wondering what PCIe does, the answer is usually "they do the hardest, fastest, highest-performance thing and they do it so well that nobody notices."
Not even close, and there are 5 paragraphs. Also, really well written.
Each PCIe lane is an independent SERDES. Its not like a parallel memory controller where they share a clock, when you have x16 lanes, you have 16x independent streams you're shoving to the device downstream.
PCI SIG has regular validation events and beyond that the lab where I cut my teeth is well renowned for hosting plugfests for cutting edge technologies.
So a lot of the reason for using some of these high end scopes/analyzers is bypassed by the testing/diagnostics functions on the serdes itself. And then of course there are a ton of signal tuning parameters which get programmed during link training.
Even modern analyzers have problems with this, and I've seen vendors that instead of even trying to reconstruct the signal at the endpoint use virtual transmission models and require one to probe at the source.
This is one of those areas that I bump into every couple years, and its frequently quite educational.
A BNC-to-PCIe-lane board with some signal conditioning would make for a hilarious oscilloscope.
This makes me smile. When I left PHY testing a few years ago SMA was no longer keeping up with the > 25 GBaud PAM4 standards coming out.
BNC was useful for PHY development in the 90s.
If nothing else it would make hardcore overclockers step up their game and start talking more about signal integrity. :)