Hacker News Reader

Comment by mechagodzilla | Hacker News Reader

I’ve done a lot of PCIe hardware development. I think there was some notion of a 32x connection early on, spec wise, but x16 is as wide as it’s ever gone in practice. All high performance PCIe traffic is controlled by DMA hardware on the endpoint, so having an endpoint with multiple logical x16 connections back to the host cpu would be quite straightforward to implement, and you wouldn’t need to worry about any of the gory details you listed. Some FPGAs are available with multiple endpoints embedded in them, for instance.

View on HN

Could you explain it like I'm five (or really like I'm a software geek and don't know this stuff) how the signals in the multiple lanes are synchronized? Is there some centralized clock that everyone is using?

The signals are skewed in predictable way, which allows recovering the ordering of transmission. IIRC, there's no common clock at all, each lane is "independent" and the ordering is applied on top.

From what I understand, at the physical level PCI-E uses an 8b10b encoding, which allows the clock to be recovered from the data. Each lane's clock might still be skewed with respect to other lanes, but there's some deskew logic that detects and corrects for this skew. I'm not 100% sure when the skew detection happens. It might be negotiated as part of link negotiation, or possibly it gets detected at the start of a packet, which has a special 8b10b symbol that's not used for data so it's easy to spot.

PCIe 3.0 actually uses a 128b to 130b encoding (I guess clock recovery got a lot better, but they also use different whitening patterns). Data is distributed across the lanes, bit by bit, and reassembled by the receiver. There's a bunch of training and link configuration stuff going on.

Once you have multiple channels of these (because 16 lanes wasn't enough bandwidth) you have the possibility that one channel gets transaction errors while the other doesn't, resulting in skewed arrival or even just missing data. So you need a software layer to deal with this, it's no longer "plug something in and you get bus level reliability," instead it's more like networking, where you have to do some reassembly as data arrives, and probably embed sequence number or other information in the data so you can identify bad and missing transfers. (Or you could say, "screw it, it's reliable enough" and wait for customers to complain; if they never do, you either made the right engineering decision or your business isn't big enough yet :-) ).

Reply on news.ycombinator.com