How PCI express devices talk

This is pretty cool, thanks for the intro.

The funny thing is this is _almost_ what I wanted the system interconnect to evolve towards back in the early 100Mbit ethernet days. After AVB and I wandered around the valley convincing the card people that you could signal over copper at 100Mbit, and they bit and started building cards, I started thinking about the future.

It was fairly clear that the hardware guys got it at 100Mbit, they could keep the packet format and keep making the interconnect faster and faster. The hump they had to get over was that the packet was the thing that had to stay the same, you can use whatever tricks you want to signal at the next speed level. So I was thinking wouldn't it be cool if they really pushed it so that it was viable for system interconnect. I half heartedly pushed for it inside of Sun but never got anywhere. At the time I was pushing for ethernet out to the disk drives (with power over ethernet so you just plug in more storage, easy peasy) but I definitely had the switched interconnect in mind as a replacement for the system bus.

So this seems like that vision, a switched network, but at a slightly different level than ethernet, it's got more end to end stuff in it.

I sort of wonder if we'll get there at some point, where the system interconnect is sort of like Formula 1 ethernet and the networking cards are like commercial car ethernet where some of the Formula 1 stuff has been trickled down to high volume stuff.

Eventually we'll run out of ethernet addresses though I bet anything we could just have a rollover, start reusing any address that is more than 20 years old (or for that matter, just have a block of addresses that are domain specific and are reused in each computer's internal interconnect).

If the author sees this message, thank you! Just the kind of tutorial I was looking for!

I'd love to see one that built on this a little and showed a packet from kernel space in one device being sent over a PCIe NIC and then bring received by another device, and DMA'd from the receiving NIC into kernel space etc. I already know that whole process more or less but it would be nice to see it and have my understanding confirmed :)

I love the comic on the right (http://xillybus.com/media/comic-300px.jpg). Never heard of this software yet. Amazing that it is part of the Linux kernel (in terms of part of the Ubuntu distribution, for instance).