Hacker News Reader

Comment by Scene_Cast2 | Hacker News Reader

One interesting thing to note is that all the high speed interconnect standards (GDDR, PCIE, USB, Ethernet) are moving (or have already moved) to RF-style non-binary communication (as opposed to bumping up clock speeds or increasing pin count). I wonder what the next steps will be for interconnects - full-blown baseband style transceivers with QAM perhaps?

View on HN

Generally the limiting factor for approaching the asymptote you observe is power. There is a figure of merit with units of Joules per bit per second that gets worse - much worse - as you start to get fancier with your modulation.

Most of the fancier schemes take advantage of the fact that traditional binary signalling has excess noise margin, ie, they were throwing away energy to start with, and they are encoding extra bits in the energy budget. But to maintain noise margins as you cram more bits in, you have to up the amount of energy per bit.

The other half is that the physical layer implementation that does the encoding and decoding consumes more energy because it's doing more computation. This also figures into the energy per bit metric if you're being honest about your comparisons (and because it is not always clear if this is included in a metric you find papers where people cherry pick numbers to make their case). This number can become quite big because the baseline of a binary tx/rx is so low compared to doing effectively a DAC/ADC and phase recovery system.

What you find is that QAM or more schemes are certainly possible, but they can consume more power than the CPU just to keep the link idle and trained. The real art is picking the implementation and developing new circuit tricks that we hadn't thought of before to wring a little more bandwidth without killing the power budget.

Would power actually be an issue, for the 200w+ GPUs these memories will likely be used with?

Power on-die is always an issue because many chips end up limited by thermal dissipation.

This is probably a stupid question but I don’t claim any knowledge here. Even though interconnect standards are moving to non-binary communication, doesn’t the signal eventually need to be converted back to a bitstream at its destination? Does this just push the bottleneck around, or do I just not understand the problem being solved? It’s almost certainly the latter and I’d love to understand more.

You’re right that the signals have to be converted back into bits at the destination. Basically, this solves the problem of pumping those bits at high speed across traces on a circuit board vs within a chip. The longer a signal has to go, the harder it is to maintain its integrity.

A serializer/deserializer (serdes) is used to convert between high-speed serial I/O outside the chip (e.g. 100 Gbps) and lower-clocked parallel signals inside the chip (e.g. 64 bits at 1.5 GHz). Using serial protocols reduces the cost and thickness of cables while parallel wires are cheaper inside the chip.

Not a stupid question - you can think of the problem by analogy with RF engineering. You have very high performance digital logic and precise clocks on the chip that you can use to encode/decode (convolve/deconvolve) bits into waveform signals and time those signals before they leave the chip at minimal latency/power expense. Once the bits are off the chip, you have no such resource and are dealing with all kinds of impedance and noise issues, which is why there are separate circuits/logic dedicated to training and calibration of the encoding parameters of the signals sent over the wire in DRAM chips.

This more complex encoding scheme is just the next level in that process, indeed moving it closer to techniques used in RF engineering.

The important distinction is between on-silicon interconnects and off-silicon. On silicon, the interconnect density is much higher so it's feasible to get high bandwidth by having lots of "channels" (wide busses). This becomes less desirable off chip (on the PCB) for many reasons. Some of them are: 1) The interconnect density is much lower, and having such a large bus becomes physically large 2) The bus is larger and less precise, which makes maintaining skew between channels more and more challenging 3) The parasitic capacitance/inductance (i.e. energy storage) of the bus becomes larger as it gets physically larger, meaning each channel needs a relatively large driver circuit (which costs expensive silicon area) and dissipates more power to drive correctly, and even more power if the speed is increased.

Increasing the symbol complexity of each channel does more than just move the bottleneck around, because it allows fewer chip to chip interconnects to carry more data.

I don't work in this regime, but as a layman I'm not convinced using full QAM for on-board chip to chip interconnects makes sense. One major advantage you natively have over the RF case is you can be easily coherent (shared clock). Throwing this away to do carrier recovery introduces a lot of complexity and potentially reduces the available bandwidth. Assuming you transmit without a carrier, can you have "baseband" QAM without a separate I and a Q signal? If you transmit an I and Q signal separately, does that not just become the same thing as two PAM-32 signals?

> On silicon, the interconnect density is much smaller

Did you mean higher?

yes sorry, pitch is smaller density is higher

There are a few drawbacks with transmitting QAM with two signals

- one needs 2 signals instead of one (2x total bandwidth) - requires each channel bandwidth to extend to to DC, which had many other challenges

If one modulates the signal to shift it away from DC, the “negative/mirror” frequencies also shift, which means now bandwidth has doubled.

A QAM signal still has double the bandwidth of an equivalent PAM one but pays for it by encoding two PAM signals.

Of course, Discrete Multitone Modulation puts QAM to shame for non-flat channels as it can adapt near-perfectly to such. Not likely to happen for high speed interconnects in our lifetime. I suspect photonics will happen first.

10G-baseT ethernet uses 128-DSQ modulation (LDPC coded). It's 128 symbols sent over 2 cycles.

These complex modulations have a huge drawback though: latency. You need to pack, ramdomize, encode and modulate at the transceiver, and undo all these + equalization at the receiver. Especially feed-forward equalization is a huge latency source.

Some numbers here: https://wavesplitter.com/technical-column/latency-and-power-...

> Cable latency is about 5ns/m, so 2.6 µs is equivalent to latency of a 520m cable. This latency might become a big issue for the HPC-based applications.

10GBASE-T is rated up to 300m cable. I don't think these numbers include 300m cable latency. It's just the PHY latency itself.

Sharp signal edges are hard to get at higher frequencies - it seems quite natural

Optical communication is using QAM so that's probably the next step.

I wonder what the next steps will be for interconnects

PCB manufacturers start offering FR-4 with strands of single-mode fiber embedded in it?

At the physical layer, digital/binary doesn't exist. It's all signals.

Technically, it's the opposite - all measurable physical things are only measurably digital (i.e. measurable in discrete units). But for all practical purposes, that threshold is low enough that it doesn't really matter for current signals transport standards.

Reply on news.ycombinator.com