Running a Raspberry Pi with a read-only root filesystem | Hacker News Reader

Running a Raspberry Pi with a read-only root filesystem

dzombak.com

218 pointsingve75 comments

Surprisingly, the article doesn't seem to mention SquashFS[1] or EROFS[2].

Both SquashFS and EROFS are filesystem specifically designed for this kind of embedded, read-only use case. The former is optimized for high data density and compression, and already well established. The later is comparatively new and optimized for high read speed.[3] SquashFS as a rootfs can already be found in many embedded applications using flash storage and is typically also combined with tmpfs and persistent storage mount points or overlay mounts.

For both those filesystems, one would build a rootfs image offline. In the Debian ecosystem, there already exists a tool that can bootstrap a Debian image into SquashFS[4].

[1] https://en.wikipedia.org/wiki/SquashFS

[2] https://en.wikipedia.org/wiki/EROFS

[3] https://www.sigma-star.at/blog/2022/07/squashfs-erofs/

[4] https://manpages.debian.org/testing/mmdebstrap/mmdebstrap.1....

I have a pi or two running openwrt, which does this naturally.

I also have a pi running pikvm. You change the filesystem by doing:

  # rw   <-- makes filesystem read/write
  # (change config, or commands making filesystem changes)
  # ro   <-- back to read-only

This is actually why I switched all my Pis to Alpine Linux since they use a RAM disk by default for RaspPis Haven't lost an SD card since, meanwhile my Octoprint RasPi has failed 4 times (mostly because of random power disconnects)

I'm a fan of Alpine too, but I want to point out that to the best of my knowledge at least, running root from RAM is trivial and is not something you need to change distros for, you simply have to add the "toram" kernel boot parameter. For example, if your entire read-only root is contained in "filesystem.squashfs", then adding "toram=filesystem.squashfs" would read all of this into RAM on boot.

I can't vouch for every single distribution out there, but I've done this successfully on Debian at least.

Very true. Before I found Alpine I used a heavily modified raspbian with a root read-only ramdisk which also worked but Alpine did it out of the box without having to modify everything

This problem seems very overblown to me. The simple answer is "don't skimp."

- Use a power supply that can deliver enough current

- Use good USB cables that can actually carry said current (a lot of people miss this one)

- Do not use cheap SD cards

- Oversize your card so that the card can properly perform wear-levelling.

I have a pair of Raspberry Pi's (2's and/or 3's IIRC) running constantly since 2015. They're running an ancient version of Redsleeve (and are on my list to rebuild but never mind that). DNS, DHCP, and a few other services run on them. They've also survived multiple abrupt powerfail events.

They log to the SD card, and yet I've never had a problem. I personally believe that the fact that I'm using oversized cards with gigabytes of free space is the key. Well, perhaps that and the fact that my personal home directory is on NFS.

Or maybe I'm just lucky? Who knows...

Not sure how things are these days, but it used to be the case that power fluctuations could kill an rpi's sd card. So yeah, you could mitigate with a great power supply and a good surge protector, but in practice I found it hard to figure out which power supplies to actually trust.

For my own deployments, I ended up doing read only boot from SD card, making use of a ram disk for temporary files, and hosting the actual os on a USB stick, which was much better shielded from power issues. Worked great, never had issues, but the setup was pretty non standard.

Chances you have any online notes on that?

Sadly, I don't think I do; I was mostly messing around with pi's ten years ago, and don't seem to have blog entries specifically on the usb disk.

The main idea, though, was to set up the os so that the boot folder was on the SD card, with root mounted from the usb drive, and /tmp on a ramfs. Iirc, I would install normally on the SD card, and then copy data to the usb stick and edit the boot files to point where they needed to. And finally, set the SD card to read only mode once everything was settled.

I think there's better options for all of this on more recent pi's, though. I think you can just boot directly from USB now, for example, instead of relying on the SD card.

Similar. I've been running multiple Pi's for years, and had issues with two.

Got a USB cable tester and found both had cables with 1+ Ohms resistance, which would cause drop to less than 4V under load!

Swapped both cables to some good ones and never had an issue since.

I always use A1 or ideally A2 rated cards from reputable brands bought from reputable stores. Not sure if they help with longevity but the additional IOPS are very noticeable.

As someone who kept my Pi 1 in a drawer for years and then bought a Pi 4 after they became available again but after building out a bit of a home lab, the internet consensus seems to be you're lucky.

The Pis are energy efficient and worth running apps where catastrophic data loss aren't a concern. I use my Pi 1 with DietPi to run AdGuard and my Pi 4 for simple web apps, like dashboards.

But the world has moved forward and there are options like the ODroid that are almost as cheap and energy efficient, without the relying on an SD card. I run my Home Assistant off my ODroid with nvme. They are all backed up to a couple different places, but my failure expectations are completely different. I fully expect my Pis to just have completely ruined cards one day, whereas I am more worried about a bad update to my Home Assistant server.

Why had you preferred AdGuard over PiHole? Just curious, didn’t follow AdGuard at all, never used it.

I looked at both, both seemed like mature products that do exactly the same thing but AdGuard had the better looking UI to me. It worked exactly like I expected so never had a reason to go back and try PiHole.

You can also use a usb flash drive

for stuff that has to be more reliable, I use openwrt (pi + ntp + usb gps + rtc => a standalone ntp server)

The last few times I used a read-only rootfs it was to build an appliance-like device which you plug in, do stuff and when its not needed can be unplugged. For this, there's no other solution.

I've been running a public facing website on my RPi 2 since 2014 and I am still on the original SD card. But, all assets and logs are being written to tmpfs, and the only thing on the SD card (outside of the OS) is the DB.

Or maybe I'm just lucky also.

> The simple answer is "don't skimp"

Yes, but then the budget then quickly rises to a point where a compact desktop such as a used Intel NUC is cheaper, on top of being far more powerful.

Depending on how the machine is being used, the Pi could be cheaper through lower electricity costs, right?

If one Pi is sufficient for the task, then nothing will be more energy-efficient. With workloads requiring multiple Pi, the used NUC wins. Your electricity costs may vary...

While they didn't use tried and proven method that's in raspi-config, it's not about skimping, it's a fundamental underlying change. Your expensive SD will eventually fail. It just will take longer than the cheap one. The question is, when will that be? If you use a read only technique, you don't have to guess.

True enough, but if something breaks with a read-only SD you are relying on logging to a cloud service.

So you reduce the chance of SD failure but introduce a risk of network failure. Of course, network failure doesn't have to be fatal but you won't have any logs to know what happened.

In my experience the chance of one of my Pi's not being able to access the network for one reason or another is much larger than a failed SD card, so I keep my logs on the card.

> Oversize your card so that the card can properly perform wear-levelling

I think this is half of the problem: SDHC does not require wear-leveling at all, I am unsure of SDXC. But regardless, the wear-levelling algorithm the SD card firmware does is completely invisible, from some vendors it might do a decent job, from some it might be a no-op, we have no way to tell.

The only way to know you're getting wear-levelling is to do it at an OS level, e.g. UBIFS stacked on MTD emulation - but there's a bunch of assumptions there which make it unsuitable for SD cards.

Likewise I have 6 Rpi doing various things all with minimum of 64gb sd cards and official Rpi power supply. Oldest running continuously since ~2014. Never had an issue. I just spend £18 on a 128gb sandisk card from a trusted seller instead of ~£4 on an small cheap one.

I too have been worrying about SD card failures in my raspberries but haven’t had a single failure ever! My cards are read/write all Debian and have been running since 2016. I hope this post won’t jinx it but my entire data centre has been running with zero storage failures despite running in the most humid and dusty basement you can imagine, though I did have other hw failures such as power supplies, rams, and ups.

There's one (not always but often enough) viable alternative to read-only root filesystems that is often overlooked. It's the possibility to copy the entire rootfs onto a ramdisk when booting. It increases the boot time and obviously disables persisting data (logs etc.), but often these are not needed anyway.

It's just a few lines of code in initramfs!

    --- a/usr/share/initramfs-tools/scripts/local 2021-11-05 12:50:23.541088057 +0100
    +++ b/usr/share/initramfs-tools/scripts/local 2021-11-05 13:02:14.483203576 +0100
    @@ -180,9 +180,20 @@
    
        # Mount root
        # shellcheck disable=SC2086
    - if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then
    -  panic "Failed to mount ${ROOT} as root file system."
    - fi
    + #if ! mount ${roflag} ${FSTYPE:+-t "${FSTYPE}"} ${ROOTFLAGS} "${ROOT}" "${rootmnt?}"; then
    + # panic "Failed to mount ${ROOT} as root file system."
    + #fi
    +
    + mkdir --parents /tmp/diskroot
    + mount -t ${FSTYPE} ${roflag} ${ROOTFLAGS} ${ROOT} /tmp/diskroot
    +
    + mount -t tmpfs -o size=6G none ${rootmnt?}
    + chmod 755 ${rootmnt}
    +
    + cp --force --archive --verbose /tmp/diskroot/* ${rootmnt}
    +
    + umount /tmp/diskroot
    + rm -r --force /tmp/diskroot
    }
    
    local_mount_fs()

and `update-initramfs -u`.

Even better, you can just put the whole root fs into the initramfs image, and never pivot to another root filesystem during boot. This also allows you to netboot a whole OS.

One pitfall of this is that the decompressed contents of your initramfs must fit within half of your physical RAM since Linux decompresses it into a tmpfs.

Or if you set rootfstype=ramfs, then you can take up to all of physical RAM, but ramfs isn't swappable.

Wait I'm not following, why must the decompressed contents of the initramfs fit within half of physical ram?

Initramfs (and tmpfs) are swappable right? As you say, ramfs is the one that isn't swappable. So if your initramfs is "to big" can't it just be swapped out?

> Wait I'm not following, why must the decompressed contents of the initramfs fit within half of physical ram?

It makes sense if you replace "initramfs" with "initrd":

> One pitfall of this is that the decompressed contents of your ~initramfs~ _initrd_ must fit within half of your physical RAM since Linux decompresses it into a tmpfs. > Or if you set rootfstype=ramfs, then you can take up to all of physical RAM

If you go for an ephemeral rw fs anyway, you can just mount a rw overlayfs on top of a ro rootfs and skip the whole copying process.

I like using Alpine for read-only operation on the Pi right out of the box: https://wiki.alpinelinux.org/wiki/Raspberry_Pi

Alpine has been great for me too.

Use an USB Flash drive mounted as a R/W filesystem for logs, etc. fsck can usually recover those f/s when crashed. The root f/s remains intact.

This is the way. Run from ram and an excellent selection of packages to boot! I have a number of pi zero and pi 2 zero running alpine.

Several mentions here of using overlay file-system via the RaspberryPi specific raspi-config options.

However, where-ever systemd is the init one can do it via adding to the kernel command line

  systemd.volatile=overlay

"If this setting is set to "overlay" the root file system is set up as "overlayfs" mount combining the read-only root directory with a writable "tmpfs", so that no modifications are made to disk, but the file system may be modified nonetheless with all changes being lost at reboot."

See man 7 kernel-command-line for the other volatile mode options.

The Pi I have powering https://hiandrewquinn.github.io/selkouutiset-archive/ has a read-only / . Everything happens in RAM, and resetting everything back to normal is a matter of unplugging and replugging the device back in.

I could do so much more with this resource, but I'm honestly quite happy with it as it is.

You can skip the ntp/systemd-timesyncd/fake-hwclock stuff if you add a RTC/coin battery to the device. The Raspberry Pi 5 has built-in RTC with support for an official battery which costs about five bucks and works well for this. Older devices have aftermarket RTC's with battery backups as well.

Theres also a kernel option to sync PPS on a gpio from a gps clock

These Pi* devices have served well running their Debian version for years.

I had a Pi3b run ADS-B (generally RTL-SDR) for many years. Dang though, all with (seemingly) random SD storage failures. Article details seems like it can help.

There are more options now though. Aarch64 ones that use eMMC.

I replaced the Pi3b with a Pine64 Rock64 running FreeBSD 14 booting from eMMC. Simple and just works with all RTL-SDR needs covered.

It's great to have options, both hw and os.

It's worth noting that you don't have to do this manually. Enabling "overlay filesystem" in raspi-config will do this for you, so that all changes to the filesystem are lost after a reboot.

I believe the overlayfs configured using raspi-config is a little different in that it presents the appearance of a normal filesystem but all changes are gone following reboot. Before reboot the changes are cached in RAM which can have consequences for low RAM systems.

I use the overlayfs provided by raspi-config.

Yes, I was missing a mention of that in the blog post and maybe a brief discussion why to do the manual setup instead.

Indeed. The post even mentions `raspi-config`, but doesn't mention `Performance Options | Overlay File System`. Makes me think that the author isn't aware of the option at all.

(The same way undoes read-only mode, but two reboots will be required instead of one.)

It is a puzzling omission.

And you can create a second, small partition/filesystem if you still need to persist some things. I used to have very little data to persist and I would mount/write/unmount every time I write to deal with possible power issues.

> This means you can run the Pi with a read-only root filesystem, which will dramatically increase the SD card’s lifetime.

I can't find an authoritative source, but some quick Googling brings up websites like Reddit and Stackexchange saying that SD cards have ~100,000 lifetime write cycles. I feel like it would be hard to exhaust that.

Is this a real problem people experience?

Yea, it's a real issue. Search for "raspberry pi dead SD card" to found report of ppl which have failing SD card every month or more often.

Most of those tend to be power issues.

Yes, it's not particularly hard to wear out an SD card (especially a cheap, high-densite one), with just the wear of writing logs. It does vary a lot depending on what exactly you are going, but it's a fairly commonly reported failure among raspberry pi users.

My website is on a pi-4 and hasn't had a hiccup in 4 years. So far, the SD-card has worked great.

How high of traffic is your website? And are you writing log files (like nginx or apache, plus application if applicable)?

I haven't had an issue with SD cards where Pis had reliable power. Where I used to live though we had frequent power cuts, of only a couple of seconds, a couple of times a day. SD card corruption was a major pain. Even in areas with more reliable power, for a 24/7 on device, occasional power outages have lead me to move away from Pis. Battery backups are, of course, an option, but increase the cost for a reliable and simple system.

It was a plague for years where power failure during a write would cause an SD card to go read only or worse.

It is, but mostly due to logging to disk instead of RAM.

That’s what setting a read-only root file system prevents in a roundabout way, so people have started to misidentify it as the solution.

A lot of people also run them with a power supply that will undervolt, switched to ones with more amps and haven't had issues since

Yes, as someone who shipped embedded systems running off of SD cards, it's a real problem.

It's not the flash endurance you hit typically, but bugs in the card firmware that cause the FTL to completely lose it's mind. Yes, even with supposedly quality brands.

About the only way I found to deal with the problem is to use "industrial" SD cards that are a little slower and pretty expensive, but the firmware takes less liberties in the name of specious benchmarks.

It’s interesting. In my embedded experience, SD cards are absolute trash, eMMC is already much better, raw NAND is somewhere in between and then actual SSDs are much better than all of these again. Yet supposedly these are all the same basic building blocks.

The "buildroot" project is perfect for your embedded system. Build a custom minimal Linux system with tiny image sizes that are read-only by default. Pick exactly which services you need and no more. Less moving parts the better.

I'm using it on 10 year old Pi's with USB RS485 dongles to get Ethernet access to solar equipment.

A long time ago I modified debian-live to use btrfs-send to copy the usb image into some kind of ramfs. So you could hypothetically snapshot & send back your current snapshot to the USB as you want.

Having a kick ass fs at your back is great. Go for it!

One of my persistently greatest annoyances about running stuff on pis has been that journald is so monolithic. There's no per-app controls. I'd love to log most system functions persistently with decent sized history (since they are very low volume), but have the main app or another app do something else. Struggling to find them ATM, but there are a number of issues opened around this way back, and all closed, saying to run multiple systemd-journal instances then configure apps to direct traffic to each, but that sure sucks to setup & configure, & is something I'd hope journald would help me with out of the box.

I have couple of memory cards (mostly Samsung) for RPi which became readonly, I keep them separately for use cases mentioned by OP.

I've since then switched to using only Toshibha memory cards in RPi & haven't had any failures.

Recently did something similar with an old weak laptop but I went all out and even avoid reads from the HDD. On startup I copy most root directories into a zram block device (maximum 4GB uncompressed, actual memory usage is 50% of disk usage), then shadow them with the copy in ram using bind mounts. Works really well, the disk is only accessed once on startup for about a minute while the system boots.

I don't usually do full read only, what I'll do is run a script that turns off stuff that does not need to be writing to the disk all the time.

Unfortunately, some software is database-oriented and likes to write to disk for every tiny thing, so the approach doesn't work with stuff like Home Assistant unless you carefully configure logging.

The basic simple stuff doesn't really cause any user-level noticable changes:

https://github.com/EternityForest/KaithemAutomation/blob/mas...

After that, I disable and mask apt-daily (The Debian auto updater), and purge dphys-swapfile.

My full set of assorted tweaks can be found here, some might not be relevant for you:

https://github.com/EternityForest/KaithemAutomation/blob/mas...

Next, I often run Chromium as a kiosk, and Chromium likes to hammer the SD card, so I set the XDG folder environment variables to make it put it's stuff in RAM. My embedded chrome stuff can be found here:

https://github.com/EternityForest/KaithemAutomation/blob/mas...

With all these tweaks, a Pi will have excellent reliability without losing the benefits of a writable FS. If you need true industrial grade perfect reliability, you probably need an industrial card and maybe true a read-only system.

I make a lot of embedded linux distress and there are a lot of ways to skin the r/o rootsf problem. Assuming you've got some bit of r/w storage, the easiest thing is making the whole rootfs r/o and bind mounting the bits you need writable.

That is what I did as well.

If you install my sbts-aru project

https://github.com/hcfman/sbts-aru

It will shrink your partitions, add new ones and install one of these and set up a sub micro second system clock and an audio recorder suitable for sound localization with a single install command. Works on all versions of Pi in both raspbian and bookworm.

If you don’t need the recorder just turn it off but you have a rootfs, a swap fs, a config fs and a disk fs.

High quality large capacity SD cards are $5-$15 now, its really not worth the extra hassle to treat them specially. Just keep decent backups

Years ago I used one of the first pis as a music/podcast player in my car. The problem was that there was no proper shutdown process, you turn of the car, and the power cuts off, so I set it up to create and mount a ram disk while booting, and do all writing to the ram disk. From the comments here, it looks like the newer pis have this as a config option now.

Cool article!

I would argue that /var is kinda explicitly for writing stuff to, so I would mount a tmpfs overlay on top of it as a more durable solution than mounting tmpfs on explicit directories inside it.

In fact, I am interested in how the procedure in this article compares to just slapping a tmpfs overlay on top of the whole system? Could be interesting to compare.

Thanks for this. I am using my RPI5 to learn about using remote computers, hacking around with and writing my tools to monitor stuff and do automation in Python.

Just putting it here in case anyone have similar blogs/articles I can read about that goes through the same journey.

Wonder can you do this for those read only publishing site using even some cloud system image. Or even GitHub based web site? What is the advantage of your own read only system. It is not changed mostly after all (or all changes only from your good self).

We use pi-builder that can produce reproducible images. Read-only is one of the default options.

Few years ago I used this script and the r.pi is still running fine. https://gitlab.com/larsfp/rpi-readonly

As it's not mentioned yet: log2ram already helped a lot in my tiny pi-hole setup.

I realized this the hard way after about 5 of my se cards were fried.

I now prefer to use overlayfs with a tmpfs overlay. Much simpler to configure and update.

years ago I did the same, I vaguely remember following a blog post to do it. I put the scripts on github and some people found it useful. There's a deb package now that does it called overlayroot.

Reply on news.ycombinator.com