NVMe SSD boot with the Raspberry Pi 5 | Hacker News Reader

NVMe SSD boot with the Raspberry Pi 5

jeffgeerling.com

117 pointsingve58 comments

> Enabling NVMe boot is pretty easy, you add a line to /boot/config.txt, modify the BOOT_ORDER in the bootloader configuration, and reboot!

and where is this /boot/config.txt stored? On the NVMe or on a SD card? (honestly asking)

On a regular computer, I don't need to touch /boot/config.txt to decide where I boot from: I set that in the BIOS/UEFI.

Basically: do you still need a SD card to begin the boot or can the Pi 5 boot directly from the NVMe SSD?

Or maybe you need to boot from a SD card only once, to change that setting (which is then stored... where?), then you can remove the SD card for good and it'll now boot from the NVMe?

Or is this something you can do without even needing a SD card?

My guess based on reading the article is you would need SD card to set things up (enable the NVMe port and set the boot order), but once setup it would no longer be required.

I recently did this on my 3 or 4 and it was exactly as you said.

This.

And apparently the firmware for the 5 might be updated to automatically detect if the external bus is wired up so you can skip that step.

USB boot without SD card has been possible since the 3 at least, I'd assume this is the same. Entire OS on one disk.

I would be totally happy with PCIe based hat that has PCIe to SATA interface with 4-6 SATA ports (with at least one of them being bootable). I know SATA SSDs are not newest, coolest nor fastest. But for home NAS setup i would find it totaly acceptable. Unlike the USB port sacrificing and "loopback" USB cable mess that is usualy used by older raspberries in order to mount SATA drives.

And where would you get power for those sata drives?

Most SSDs are 5V only, so I assume Orange Pi users just get an adequate wall wart.

Sata port does not provide power.

Sata has a standardized data port and a standardized power port. The power port usually has multiple voltages connected which would complicate things if they mattered.

I have a radxa jmicron hat on a pi4 and I did a zfs 4 disk set-up precisely so I could move to Sata on pci-e and move os with these drives. A cm4 carrier board with data or pci would do, but no matter what I too want to stop using usb to do things.

The zimablade looks interesting: 2 sata on board and a full pci-e slot, Intel quad cpu, fanless.

Hmm but that hat connects to the Pi over usb?

When will we get a Pi for storage that doesn't need to use usb?

Edit: i took a look at the zimablade in spite of the atrocious web 5.0 site with silly animations getting in the way of getting to the specs.

Nice. 2 x gigabit ethernet when?

> Hmm but that hat connects to the Pi over usb?

Yes. It's all I could get at the time. I chose zfs to allow me to move disks to true sata over pci when I can.

The 5 appears to allow a pci bridge to be used. It's possibly my next move, I've had the radxa hat for a couple of years now.

If you want a NAS, buy a NAS. I really don’t get people’s desire to take an underpowered single board computer and try to turn it into a useful server. There are absolutely places where a Raspberry Pi makes sense, those places are where you’ve got electronics you want to interface with complex software that can’t viably be run on a smaller MCU.

> If you want a NAS, buy a NAS. I really don’t get people’s desire to take an underpowered single board computer and try to turn it into a useful server.

Off the shelf options tend to have awful software, pis have good power consumption, and as of the 5 is it really underpowered?

All computers are NAS if you stick S on them and A them to an N, the Raspberry pi is no different, but it's got a great advantage in that it's small and relatively power efficient. I turned a Dell Optiplex USFF into a NAS (https://willj.net/posts/fitting-two-hard-drives-and-an-ssd-i...) but in the same space I could easily fit 4+1 disks instead of the 2+1 if I was using a far smaller motherboard.

As for being underpowered, again it depends what you are doing with them. For my NAS (basically long-term storage of stuff I don't access much and tiemmachine backups) there would be plenty of CPU, the biggest problem would be memory.

*edit* Just to be clear, the Pi has some massive drawbacks as a NAS at present because of it's significant lack of disk IO, but there's nothing inherent in it's form factor that makes it a bad NAS. If I could have plugged 2+ HDs into one in a performant way I'd likely have done that rather than the Dell Optiplex shenanigans.

Pi has superior long term support and even better community support. Loads of options for upgrades and extensions should you need one in the future. Also once i decide that i don't need that NAS anymore, because the pi would be obsolete or slow in the future, i can just take that pi out and use it for some other DIY project.

An underpowered single board computer is plenty for a basic NAS. Certainly a more expensive and more powerful single board computer would make a more expensive and powerful NAS, but a lot of people have modest needs.

A consumer NAS for most people is like, the least demanding device ever. All it needs to do is stream video at most, probably not even 4K, maybe not even that if you're just doing backups.

Buying an SBC to then use extra modules (pcie, camera, etc.) sounds like defeating the purpose of the 'single board' computer. Interesting experiment nonetheless.

It is reminiscent of the 1980s micros that inspired the Pi; there was a thriving scene - commercial and homebrew - of plug in modules. If you think of the Pi in that context rather than an SBC, it makes sense.

> Buying an SBC to then use extra modules (pcie, camera, etc.) sounds like defeating the purpose

Agree, except cameras are often more practical or flexible to buy separately.

If extra modules aren't a purpose of the RPi then why does it always come with GPIO pins? The point isn't to only have a single board but to have a simple base that's easy to build upon.

Single-board computers without any extensibility are primarily sold by Apple. Very different target audience and purpose.

Not really. I like the idea of building NAS where i can upgrade the RPi5 to RPi6 when it's released. It's not like i need everything to be "single board". But the fact most of the important components are on that single board, means the thing can be quite small (even with few extra project-specific peripherals attached)

It is exactly the opposite. Raspberry pi are created for learning and experimentation, and hobbyists too. There is an entire industry based around using it with expanded components.

Happily streaming music with an rpi 4 plus HiFiBerry DAC extension hat. What would you suggest I should have used instead?

RPi, librespot, USB DAC with studio monitors. My main remote audio stream system at home that anyone on homenetwork can use.

What difference does it make using a USB DAC as opposed to a plug-in hat DAC? How is the RPi still meaningfully "single-board" if you have to plug it in to a USB peripheral?

Not sure what you mean? Why limit to single-board?

The comment I was originally replying to said adding on peripherals defeated the purpose of a single-board computer, which I disagree with.

yeah the costs add up and you could have just gotten an old android phone at that point

Let's ignore the concept of Raspberry Pi hats for a second and instead look further back.

Nearly all SBCs have been used as simple platforms to interact with extra peripherals. The idea that SBCs shouldn't support peripherals is new, and arguably pointless. Want a computer with no peripherals to just run code? Pay a VPS provider.

https://en.wikipedia.org/wiki/Single-board_computer#History

The Ardunio from 20 years ago was designed to use hats. It has always been this way.

> (...) sounds like defeating the purpose of the 'single board' computer.

What purpose do you think it is? Why do you think people spend their cash on a Raspberry Pi or any other RPi clone?

Well, I was aiming to the NVMe storage mainly. To me that point is valid (as in, SD cards are unstable), but I draw the line. I think microdesktops with passive cool down are better suite for many applications, specially if you're looking forward to having an SSD. That was what I had in mind. Still, SBC with extendability is great for Diy and many IoT applications. Indeed.

It's just a motherboard with the CPU soldered on, it's not really /that/ much different from my Intel desktop in the sense that it's a general purpose computer that you can plug other things into. Focussing on the S bit of SBC isn't really helpful.

Ultimately a Pi or other SBC is a pretty nice size for using as a dedicated and specialised device for lots of things. I've got one plugged into my boat to monitor boat related stats, I've got one that does logging and graphing of environmental stats around my house (and runs Octoprint). Both running SSDs, and though I could for sure run these services on a larger "proper" computer there's really no need to take the space and power.

Running one as a NAS would be pretty sweet because it would be pretty small compared to my smallest NAS which is a Dell Optiplex USFF machine (https://willj.net/posts/fitting-two-hard-drives-and-an-ssd-i...)

It is a single board computer. Are you adding daughter boards? No? Even your phone has separate storage pieces. Adding storage has never broken the concept of a single board computer. I'm not sure why it would now.

I wish the Pi foundation would just fix their OS so it doesn't need an SSD.

It runs just fine with about 200 lines of tweaking to protect the SD from excessive writes.

But then, there are a few well known apps that seem to break SDs anyway, like Home Assistant, and they seem to not be interested in properly supporting reliable SD card installation, so I guess SSDs make sense.

I don't know what irritates you so much. Nowadays NVMe SSDs are cheaper or the same as SD cards per unit of capacity.

Not at the low end of capacity they aren't, and they're also physically bulky and make cases a lot harder. Plus whatever hardware they had to add to do the interfacing costs money.

Their focus is education and industrial controls or signage, I'm not sure why they're so heavy on the ultra high performance stuff lately, there's lots of stuff that money could have been spent on that would be a little more relevant, like onboard battery backup circuitry, USB-C video instead of those horrid micro HDMIs, random cool stuff like LoRa, an onboard Qwiic connector, a physical switch or button and NFC for easy setup, an RGB status LED, an onboard FeRAM chip, etc.

They're not desktop PCs or high performance computing devices, why are we trying to make them do that?

There's great dedicated hosting providers for Pis, such as https://rpiservers.com . The big downside seems to be sd card wear; a major reliability concern. The idea of an rpi5 with PCIe 3.0 NVMe is very very alluring as an entry level point of presence online.

The price point would need to be better than rpiservers, imo. $10 for a pi and external powered USB drive is not compelling enough; a shared vps is probably better/cheaper. However their $7 for a pi and unpowered USB drive does seem great, and there are some really amazing USB drives. Doing that for a rpi5 would be awesome.

It'd be nice if the pi could do 10Gbps on one port or two 5Gbps ports, so we could plug in a fast USB drive. But even with only 5Gbps, that's still around the same perf as the officially supported PCIe 2.0 speed.

Why would anyone go through all this trouble just to get an nvme running on RP5? I get the fun in complexity but just not worth the effort. Sure you are learning through the process but I hate that you have to jump hoops just to get that option available when they could have done so at the design stages. I love my raspberry pi 4. But I also enjoy my Dell Micro PC and HP Elite. They make awesome media centers and much more. Regardless of the opinion, I appreciate the information shared.

The plan to turn Pi into NAS moves to the top priority again!

Great write-up once again Jeff. Thanks!

I enjoy his "voice" (in the sense of presentation style) a lot. I originally watched for a few tech videos on the Pi but last night watched him paint his office and I'm not even sure why I did that, but I enjoyed it! Looking forward to this one.

What SD card do you need for maximum speed on the Raspberry 5?

Is there a reason why the Raspberry didn't support A2?

Yeh I’d like to know too. I have a Pi5 on pre-order.

I’d also like to know how much faster the NVMe SSD drive is to the fastest microSD

A lot. Fastest microSD through UHS-I interface tops out around 100 MB/sec. NVMe SSD tops out at 450 MB/sec at Gen 2 speed, and 900 MB/sec at (technically unsupported, but working) Gen 3 speed.

So is the UHS-I support hardware bound?

Or could we get those speeds on Raspberry 4?

Hardware; Pi 4 only used SDR54 or something like that, half the data rate as Pi 5 microSD interface.

Do you know if Pi5 has SD/A2 support?

I would also think that random read/write performance (throughput/IOPS and maybe latency too? just a hunch...) on SSDs will be miles ahead of microSD and likely represents impacts on real-world/OS scenarios much more than sequential numbers

It's almost like you guys are allergic to eMMC. Crazy stuff.

Trade-offs, wild - right? Performance, expandability, and so on

As the end users I'm not sure how much criticism they deserve for not embedding the storage.

If they have to buy something extra... why not go with better performance, reliability, and expansion choice? An eMMC hat or something is exactly that, forever, until it fails.

Adapting expandable SSDs deals with all of this

Nah, it seems like it always comes down to price. Rpi people love that $30 savings and then spent the last decade trying to figure out how to make uSD reliable.

Then again, you never have to worry about unbricking devices.

I’d love a small board that offered networking, NVMe SSD slots etc that I could plug a Raspberry Pi Compute module with eMMC into

It's out there, but you won't like the price. And it doesn't use an RPi.

The Home Assistant Yellow does that, Ethernet port and NVMe slot. If that exists, then there is certainly a less specialized one. I found a few others, the Piunora Pro seems to be the smallest and $50 but no Ethernet.

Reply on news.ycombinator.com