Adafruit's Read-Only Raspberry Pi | Hacker News Reader

Adafruit's Read-Only Raspberry Pi

raspberrypi.org

78 pointstdrnd43 comments

I'm operating a digital signage service for the Raspberry Pi (https://info-beamer.com/hosted). When we started designing our Linux based custom operating system we also decided to go with the read-only approach. So far we haven't had a single device go offline due to SD card/file system problems. It was really the only sensible way to do this as you never know when/if devices get turned off.

We also handle system updates by using a A/B partitioning scheme, so you can even turn off a device in the middle of an update without breaking it. The additional benefit from that is that all system updates are atomic and self-contained, so you always have the newest version without any potential leftovers from the previous version (or the previous version if the update got interrupted). Customers that have installed our first version 3 years ago now run the exact same OS that we released today. It makes it really easy to provide support.

For obvious reasons we also need a read/write partition to store downloaded assets. We treat this partition as volatile and have an easy way to automatically verify and repair content if anything gets corrupted for whatever reason.

If you want anything running reliably on a Pi, I feel this is the only way to do this.

Have you tested unexpected powerdowns durring writes? Unless SD card is specifically designed to survive this, I would guess you might get corruption anywhere on the card, despite writing to specific area of the card, due to wear leveling algorithms. It might only become an issue when the card was used for a while, so that all flash blocks were provisioned at least once.

Not 100% of course, but so far we haven't encountered any problem. We also checksum both the operating system and all downloaded data periodically so we can catch and repair any corruption. Of course this won't help if the system doesn't boot any more. So far we haven't had this problem.

In addition to the system part being read-only, we also try to minimized writes to the data partition. We're using some of the ext4 settings to make sure writes happen less frequently as we're perfectly capable of recovering from lost data after a reboot.

I actually implemented pretty much the same layout (custom kernel, A/B partitions, but one each for the OS and the player software) for a similar use case (background music player). We used AUFS for partition layering. In particular, our setup was as such:

OS: SquashFS file system (ro) -> Config partition (ro/rw) -> tmpfs partition (rw)

Player software: SquashFS file system (ro) -> Data partition (ro/rw) -> tmpfs partition (rw)

The config partition is mounted read-only, all changes are written to the tmpfs partition. We then use a script to bulk write desired files to the Config partition (i.e. remount rw, copy files, remount ro). This limits the write time to a fraction of a second (these are only small text files).

As for the data partition, we remount the partition rw when we're updating the music files and check for filesystem consistency on bootup.

We hadn't had a single system fail because of the SD card in over 2 years.

Neat! We're also using a SquashFS root file system. It's only 30MB total at the moment.

Right now there isn't a way to do remote system configuration changes (through a UI, it's doable from a command line). Most changes happen through the "apps" deployed on those devices and those configuration is treated as volatile and might get lost during power outages (but then repaired later). If we decide to allow modification of the few system settings we have, I might treat them as a new A/B cycle and recreate the complete system with the changed settings on the other partition. That way I also get automated fallback if anything is misconfigured and (for example) the device fails to connect to the new WiFi.

This is pretty neat. Have you released anything used to build and manage the Pi aspect of this?

I haven't. So far it's really customized for our use case. If you're interested in a smallish self contained code that is based on Go, you might have a look at https://github.com/gokrazy/gokrazy

Does this compare or relate to Ubuntu Snaps? Obviously not the RO portion, but the handling the updates et al?

Otherwise this is really good to know. I guess I don't run RPi's enough because my lazy unplugs haven't been an issue...yet.

It's not based on Ubuntu Snaps. It's a custom development. It predated Ubuntu Snaps by I think 2 years? I don't see any obvious reason to switch to Snaps now, since it's all been really reliable so far. The downloads are done using zsync. Usually a system update is less than 5MB total.

My old employer used the same A/B scheme for upgrades, and it did indeed work great.

That doesn't help against corruption on the SD card itself from unexpected power down or cheap power supplies... especially the latter has bitten me more often than I can count.

I'm a bit confused. The article says "They’ve shared a script which forces the Raspberry Pi to run in read-only mode, so that powering it down via a plug pull will not corrupt the SD card." - Is that not the same as an unexpected power down?

That's two different sources of corruption. The shared script prevents that the SD card is being written to - which means that the filesystem won't get corrupted due to half-written or inconsistently written data. The other source of corruption is the SD card's microcontroller which does not know that the SD card is being used in readonly mode and may do stuff like housekeeping, garbage collection etc. nevertheless - which might be interrupted by a power-out.

If you have the OS level setup for read-only access, can't you also flip a physical write-protect switch on the media at that point to defend against even that?

The write-protect switch on SD cards is not a physical break of a write line. It's merely a signal to the host saying not to write. A malicious or badly written driver can disobey it.

Not only this - microSD cards don't even have a WP switch any more, and on full-size SDs the thing is more unreliable than worth the trouble...

I would have guessed a signal to the local microprocessor to stop any writes as well. Oh well.

So would the solution be for the host, or some SD adapter, to provide a super-cap to ensure smooth shutdown of the SD unit as a whole?

A supercap is indeed one way go to, but it's useless without a decent brownout detection circuit that doesn't freak out every time you lose an AC halfwave (happens more than you think).

Hmm. A supercap combined with a diode on 5V and a line to a GPIO from Vcc to provide enough time for the OS to shut down... like this

    (RPi GPIO)        <-----------(resistor to 3v3)-|
    (RPi microUSB in) <-------------(diode <-)------+---- Vcc
                       | Supercap
    (RPi GND)         ----------------------------------- GND

When the supercap is huge enough (or, for what its worth, use a powerbank with passthrough capability!), this should work - when the GPIO goes low, the system can initiate a safe shutdown, and especially it can signal a "shutdown" signal to the microSD.

Not only that -- the part on the SD card isn't even a switch. It's a plastic nub that will (hopefully) activate a leaf switch inside the SD card socket.

It's bitten me quite a few times as well. Maybe I will try this read-only approach and see how it goes. I agree that it won't help against corruption on the SD card itself, but I suspect that most of the times the SD card has been corrupted, it has been due to power failure (or bad power supply) combined with writes at the same time.

A proper button to trigger the 'shut-down' sequence would be better.

it's actually relatively easy to add yourself:

https://gilyes.com/pi-shutdown-button/

You can't rely on users to actually do this every time. The more robust you make your system the better.

On the contrary, a sudden power loss is an exceptional case. Not having a button is just Raspberry Pi's (generally poor) hardware design decision, along with countless others relating to the power system and usability. Making your system entirely read-only _for this particular purpose_ is a hack, not a solution.

Of course it's a solution. It solve (among other things) the problem of unexpected power loss. You can't design your system for the happy path and hope for the best.

Do you turn off your computer by pulling the plug? Anyway, I was just reminded of 90's. The times before ATX.

Of course not. Because my OS isn't specifically designed for that. If it was (like my router): Of course.

Actually, this is not a poor hardware design decision. It makes sense for certain systems to have a power buttons, other systems don't generally use one (e.g. routers). Power down strategy is left as an implementation for the user. Read-only systems have many advantages and I wouldn't consider them a "hack".

You can't, but it's useful nevertheless. I have several Orange Pi's running as servers for various things, having HDD connected over USB, etc. I'm very glad it has a micro switch that can initiate a proper shutdown of the system.

I certainly wouldn't want to just unplug them from power.

From an embedded perspective, it's ridiculous to expect that a computer will be harmed when simply pulling its power. The phenomenon is due entirely to shoddy engineering/programming that pays no attention to transactions. Babying systems just encourages it to continue.

Rather than superstitious behavior, we need cheap test jigs that can (for example) validate newly-purchased SD cards so that defective ones can be immediately returned.

Actually, from an embedded perspective, almost all SD cards do not guarantee that they don't corrupt when you kill power during a write. This is because of the internal workings of SD cards and completely unrelated to shoddy engineering/programming.

Those "internal workings" you refer to are the shoddy softwares on the SD card controller! It's certainly possible to implement transactions on flash.

It's good to see a nice easy script to do this. I can confirm that mounting the SD card as read-only works great. For a couple years, I used a Raspberry Pi as an AirPlay receiver hooked to a set of speakers, and whenever I was done listening, simply turned off the power. I had / mounted read-only and never had SD card corruption.

Reminds me of the "IPE" project, Industrial Perennial Environment. I've successfully used that for a few things, some of which is still running without corruption for years now.

Good recovery from power fail ought to be the normal case. It is for just about everything else. What's wrong? Does the thing need better shutdown management hardware? Is it powering down without going into reset before the hardware starts malfunctioning? (The Motorola 68HC11 was notorious for that.) Or what?

BeagleBoard has this problem, too.[1] They have a power managment IC and shouldn't have the problem, but it's not working right.

[1] https://elinux.org/Beagleboard:BeagleBoneBlack#Improper_Powe...

Isn't this how Alpine Linux [1] normally works?

[1]: http://alpinelinux.org

You can write changes to the card with a special command.

yes, but running from RAM is the default and thus safe to unplug at any time IIRC...

> The only fix? Wipe the SD card clean and start over, waving goodbye to all files you didn’t back up.

> making your Pi read-only is irreversible

I understand the article is intended for people not very experienced with computing and hardware, but come on. The only fix? Irreversible? Scaring the readers bit too much, IMO.

Would it be possible to have a laptop or desktop that spends most of its time operating in such a Read-Only mode?

From what I've seen, the systemd developers are interested in going into this direction. They established /run/ as a tmpfs that is available very early during boot, and built systemd such that it never needs to write to a persistent filesystem during its operation. (It does, by default, write logs to /var/log/journal, but you can switch it to non-persistent storage at /run/log/journal instead.)

Many programs can be made to work the same way. There are still plenty of programs that need to write longterm state to /var/lib, but you will probably need a writable partition anyway for /home, so you can just reuse that partition for /var/lib as well.

OS updates would then be handled via an A/B partitioning scheme, where you have two copies of the system partition, and when an OS update is ready to be installed, the inactive one is mounted read-write and the update written onto it.

OpenELEC has it by default, nothing new or revolutionary.

Reply on news.ycombinator.com