Raspberry Pi PCIe Devices | Hacker News Reader

Raspberry Pi PCIe Devices

pipci.jeffgeerling.com

190 pointsgeerlingguy58 comments

I love a good “PCI Challenge”, where someone tries to see how much functionality they can wring out of PCI devices in a weird situation. Linus Tech Tips has a bunch of good ones:

- How crazy of a device can you use with a server motherboard’s PCI hot-swap BIOS feature: https://youtu.be/YigN2mkQMPc

- How many PCI extenders can you use with a high end graphics card: https://youtu.be/q5xvwPa3r7M

Ah, but have you ever run PCIe over 115200 baud serial? :)

We did that for the PS4 hardware experiments, and that also involved a PCIe hotplug capable motherboard, as we had a PS4 southbridge patched in as a PCIe device on a normal PC, and every time we powered on the PS4 that was effectively hotplugging it. Fun times.

These days PCIe hotplug is a consumer technology, it's called Thunderbolt (it's hotplug at the logical layer, not physical PCIe lanes, but to the OS/software it's about the same).

PS4 talk including the PCIe story: https://youtu.be/QMiubC6LdTA

I read the first sentence of this post without noticing your username and thought, “wait, wasn’t that marcan?” — of course it was :) It’s a good time to rewatch that classic talk (and probably some other old ccc talks, there’s a wealth of great ones.)

This is something that I miss about forums of yore and sort of hate about HN, Reddit, and their ilk: the deprioritization of a post author's identity. I miss being able to scan through a forum's post listing and readily pick out names of posters I recognized after being involved in the site for a while. It was an additional signal of post quality (or lack thereof, depending on the user). Here, the username is hidden in a sentence of status text, on the same visual level of the number of minutes ago the post was made, all of which is rendered in a light grey font. It's clearly intentional, and I can't for the life of me figure out why the forum owners don't want users to be able to build a consistent identity.

I think it’s always been that way on HN—it could just be arbitrary, not a deliberate choice. I have noticed around here that if you indicate you might use someone’s identity or history as a signal of quality, you’ll get a lot of “argument by authority” backlash. (Except in the cases of someone who built the thing you’re talking about, and tptacek. :) )

On the flip side, on HN I like that the author is kind of greyed out. I never notice it until after I read comments if I look for it, so I generally don't even notice the OP or a comment author.

Other sites like Reddit add little icons and highlights in many places that make the username much more prominent.

I remember that talk fondly, thanks for all the fun!

My first thought was "can you boot from a PCIe SSD?" (looks like the answer is "no" from what I see). I wish there was a RPi with 16GB or 32GB of storage soldered to the board. Where I work we're using Raspberry Pi devices for industrial use and corruption of/wearing out the microSD card has been something we've wondered about. We could just buy premium microSD cards but having a "proper" SSD soldered to the board would be very helpful as well.

There's the compute module...32GB of eMMC is an option.

https://www.raspberrypi.org/products/compute-module-4/?varia...

And in some good news on that front, the eMMC storage on the CM4 is twice as fast in many cases as older generations of Pi CMs, as it uses the full 8 bit eMMC bus. It's finally faster (for random IO at least) than any microSD card I've tested.

We've had good success with making SD cards read-only. Of course this means that any data needs to be sent to a service for saving... YMMV.

+1 to this. I've seen this done as well, and it works marvelously. SD cards last half of forever if all you're using them for is a boot device.

It depends on what you are needing to optimize for, industrial use is a very broad category. If you’re caring about the SD card’s endurance for reads/writes, I recommend trying the “high endurance SD” cards. I’ve had good luck with them having continual read/write activity over long periods. If however your concern is mechanical failure due to vibration, stay away from the SD card entirely. Pretty much guaranteed it’ll fail if it’s a high mechanical stress scenario. You’ve got two good options there I can vouch for, use a Compute Module w/ hard soldered eMMC or boot from USB instead. The compute module route is the best in my opinion, but doesn’t always fit the needs, and for those that don’t I recommend USB booting.

Even though you cant "boot" from a PCIe SSD or SATA disk (yet) you can simply stick the boot partition on the sd/emmc and then pull root from the pci attached disk. Kernel might load a little slow but user space should load quite fast.

If the system can see the PCIe SSD, you could always boot off the microSD card but use the PCIe SSD as the primary storage device. Definitely worth testing imo.

There are MicroSD to CD card adapters out there. A bit clunky but they allow you to use full sized SD cards, which might be more durable

Please add m.2 lte and 5g modules with sim slot. They make good for IOT projects. I have seen people doing projects. Two vendors are quectel and Sierra wireless based off Qualcomm radios.

It looks like the MCUZone CM4 4G mentioned here [1] is intended for just that purpose! I would be happy to try out a chipset provided it's not too expensive, and (ideally) it has good Linux driver support.

Drivers seem to be the dealbreaker for a few cards, if they rely on PCI express features the Pi OS or SoC currently don't support (like BIOS, or sort-of MSI-X support).

[1] https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

Those need m.2 B key slots which are not compatible with m.2 M key NVMe disks. This is why I'm not crazy about the m.2 "standard".

To be fair the same thing can be said about its predecessor the miniPCIe slot.

True but that was only really for the msata version. m.2 has about half a dozen keys as of now. Like USB-C it tries to cram too many mutually incompatible electrical interfaces into one plug.

Where do we recommend new PCIe devices for you to test?

Have you tried Mellanox Infiniband and RDMA?

Check out the linked GitHub repo: https://github.com/geerlingguy/raspberry-pi-pcie-devices — for any card I'm actually testing, I have an open issue. For proposals or discussions around a certain topic, there are some ongoing conversations in the Discussions tab.

Specifically concerning Infiniband, check out this issue: https://github.com/geerlingguy/raspberry-pi-pcie-devices/dis... - I'd be willing to test, but currently I have a bit of a backlog of other gear I'm getting through testing :)

Wouldn't you saturate the bus fairly easily with a Mellanox card?

I can't imagine the experience would be optimal.

From all my testing, it looks like the maximum throughput you can get on the entire PCIe x1 gen 2 lane on the CM4 is about 3.2-3.4 Gbps.

But that's still better than the current maximum of 1 Gbps over Ethernet, or the ~3 Gbps I have tested through a USB 3.0 card.

what??? I _did not know_ PCI-E was exposed! I literally just had another discussion on HN a few weeks back when someone posted a 3d printed PI Cluster, and everyone lamented how lame storage latency was on a PI. I guess we're all pretty ignorant of the current offerings!

This is pretty recent. The USB controller on the regular Pi 4 was sitting on the PCIE lines, so until the CM4 came out the only way to get to PCIE was to lose USB support and solder wires onto the traces on the PCB. A couple people did it but not many. Most people could get by with a USB 3.0 accessory and didn’t need to cut up their Pi. But now that the CM4 has a PCIE slot on the daughterboard it makes sense to see what is compatible.

Ah, that was going to be my followup question. I'm having trouble finding a block diagram of the CM4 (and RPI4), if you happen to have a link, that'd be incredible.

* Also, how the Ethernet PHY connect to the CPU and what are the implications?

Go back and double check! kingosticks replied to you with exactly this link.

It's a shame there isn't something between the Model B and the Compute Module, like the CM but with a built-in PCIe header say.

There have been a few popular submissions using multiple Pis in clusters recently, really the only reason not to use CMs there js the complexity of the (limited) IO that is needed, and the CM development board (for each one) dominates the cost and would be motr expensive than Model Bs.

There are a few addon boards (for $15-30, most shipping after the new year) that take the CM4 that extra mile: https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss...

I'm most excited about a few with mini PCIe or M.2 slots on the bottom.

That's why I use RockPro64 instead, they have PCIe for a long time.

It might be out of scope, but:

Can you boot from the m.2 SSD? Do you have benchmarks for m.2 SSD?

This would make my day, as all i know of is some USB-dongled SSD that (back then) could not be booted from. Sry, if this is already standard today, i am a bit out of the loop.

Follow https://github.com/geerlingguy/raspberry-pi-pcie-devices/iss... for progress. You can't boot directly from the SSD without a microSD card, but it (or SATA support for boot) may come soon... just a question of priorities for the Pi Engineering team.

Do you happen to know if booting from the SSD is limited by the closed source bits of the Pi system or is this an effort the open source community can possibly contribute?

Would a chip (such as a CPLD/FPGA) implementing mmc/sd be an option? It could host a chainloader with PCIe support and serve as a kind of rom.

Also, is there any work on kernel level x86 emulation on arm for expansion roms?

I imagine it's limited quite a bit by the odd way the Pi boots. It's the GPU, not the CPU (or a BIOS type setup), that initially runs the bootloader.

its mostly that the bootloader has to be patched to see the devices, this is no easy feat as it would bloat the bootloader pretty hard. a chain boot of MicroSD to SSD is best at this time.

Closed source firmware bits, unfortunately.

I'd be interested if Thunderbolt 2 or 3 cards could be made to work with the Pi.

Telephony cards (Sangoma/Digium) would be interesting to try as well, though if you were trying to run FXS ports you would need an additional cable and PSU to supply 12V to the card.

Graphics cards presumably need an external power supply?

Yes; thus the testing of the powered external 16x risers (see the last section of the cards).

Is there a PCIe to ISA adapter? I'd love to hack some old cards from the 80s or is there any other way to connect ISA cards to RPi?

IMHO PCIe (and SDRAM) should be the only interface exposed by a modern CPU. On-board graphics, audio and NICs usually are shitty anyway. SATA is not relevant any more - we better plug an NVMe SSD into PCIe (or get a controller if actually needed). Full USB-C (or ongoing USB-4) means Thunderbolt which is PCIe. Why waste silicon on anything but PCIe then?

> On-board graphics, audio and NICs usually are shitty anyway.

Onboard graphics, audio and NICs are perfectly functional and more than enough for most people. Both in work environments and at homes. They've not been in a shitty category for years now. They won't break any records, but they will display websites at high resolution, play clear music, and provide more bandwidth than the link to the internet can handle.

Just look at the number of Lenovo's AIOs (and similar solitons) in offices.

> Full USB-C (or ongoing USB-4) means Thunderbolt which is PCIe.

USB 2 is a fraction of the price. (And power and board space)

I really love the pushback here. Yes how awesome it is you are privileged enough to always be able to acquire the best of the best and look down on other "inferior" devices.

But 99% of the rest of us just need things that work and allow us access to good performance.

I actually use everything built-in because what I use are a decade-old business laptop, an office desktop and a RaspberryPi. But going to buy a good desktop (as I used to have 2 decades ago) I would rather buy a good mainboard with as many PCIe slots as possible (and no built-in devices if this were possible), the cheapest discrete cards for everything and upgrade them as I see fit. I doubt the cheapest/used discrete card for anything is going to be worse or more expensive than a built-in non-removable version.

> play clear music

Then why are there so many audio hats, let alone USB audio dongles used even in the RaspberryPi world?

I believe the GP's post was more general in proposing PCIe attachment. Though you are correct is asserting that stock Pi audio is junk.

That feels really counter to what I see. Maybe five years ago I'd have agreed with you, but now-a-days on-board graphics, audio and NICs are way better than they used to be.

Don't get me wrong, I still run a discrete GPU for gaming, but for a lot of people onboard GPUs provide a good experience, and often much better power efficiency.

Power and convenience. Power usage by PCIe is a factor for mobile devices. And convenience, in that "shitty" onboard graphics, audio, and NICs are good enough for 90% of the use cases for most devices.

> Why waste silicon on anything but PCIe then?

Power. PCIe is a fairly power hungry link. On-die parallel connections are incomparably more efficient.

Special low power links like MIPI PHYs also have their place.

USB is high enough speed that you will get quite a bit of overhead from USB>PCI>memory route, and back.

Audio is a particular pita, network too needs as close, and direct as possible connection to the CPU to get maximum performance.

In many cases, these all can use the same pins anyway. Like on some NXP SoCs I've worked with, you just had a bunch of SerDes lanes that could be assigned to any number of functionalities (SATA, SGMII, PCIe, etc.).

Massively depends on the use case. Why waste board space, assembly complexity and power on PCIe when a SoC does just fine?

Other than the integrated graphics that are directly part of modern Intel CPUs, everything else you mention is typically internally connected by PCI-e. Onboard USB ports lead to a controller which talks to the CPU via PCI-e, for example.

PCI-e lanes are a limited resource on a CPU and really fast. Dedicating one to a keyboard, for example, is probably inefficient.

I have heard (don't know too much about it) some Intel NICs are able to place network packets right in the CPUs cache for maximum speed, this probably can't be done via PCI-e.

This comment reeks of privilege

GPU yes, rest no.

Reply on news.ycombinator.com