Comment by dividuum | Hacker News Reader

Comment by dividuum | Hacker News Reader

I'm operating a digital signage service for the Raspberry Pi (https://info-beamer.com/hosted). When we started designing our Linux based custom operating system we also decided to go with the read-only approach. So far we haven't had a single device go offline due to SD card/file system problems. It was really the only sensible way to do this as you never know when/if devices get turned off.

We also handle system updates by using a A/B partitioning scheme, so you can even turn off a device in the middle of an update without breaking it. The additional benefit from that is that all system updates are atomic and self-contained, so you always have the newest version without any potential leftovers from the previous version (or the previous version if the update got interrupted). Customers that have installed our first version 3 years ago now run the exact same OS that we released today. It makes it really easy to provide support.

For obvious reasons we also need a read/write partition to store downloaded assets. We treat this partition as volatile and have an easy way to automatically verify and repair content if anything gets corrupted for whatever reason.

If you want anything running reliably on a Pi, I feel this is the only way to do this.

Have you tested unexpected powerdowns durring writes? Unless SD card is specifically designed to survive this, I would guess you might get corruption anywhere on the card, despite writing to specific area of the card, due to wear leveling algorithms. It might only become an issue when the card was used for a while, so that all flash blocks were provisioned at least once.

Not 100% of course, but so far we haven't encountered any problem. We also checksum both the operating system and all downloaded data periodically so we can catch and repair any corruption. Of course this won't help if the system doesn't boot any more. So far we haven't had this problem.

In addition to the system part being read-only, we also try to minimized writes to the data partition. We're using some of the ext4 settings to make sure writes happen less frequently as we're perfectly capable of recovering from lost data after a reboot.

I actually implemented pretty much the same layout (custom kernel, A/B partitions, but one each for the OS and the player software) for a similar use case (background music player). We used AUFS for partition layering. In particular, our setup was as such:

OS: SquashFS file system (ro) -> Config partition (ro/rw) -> tmpfs partition (rw)

Player software: SquashFS file system (ro) -> Data partition (ro/rw) -> tmpfs partition (rw)

The config partition is mounted read-only, all changes are written to the tmpfs partition. We then use a script to bulk write desired files to the Config partition (i.e. remount rw, copy files, remount ro). This limits the write time to a fraction of a second (these are only small text files).

As for the data partition, we remount the partition rw when we're updating the music files and check for filesystem consistency on bootup.

We hadn't had a single system fail because of the SD card in over 2 years.

Neat! We're also using a SquashFS root file system. It's only 30MB total at the moment.

Right now there isn't a way to do remote system configuration changes (through a UI, it's doable from a command line). Most changes happen through the "apps" deployed on those devices and those configuration is treated as volatile and might get lost during power outages (but then repaired later). If we decide to allow modification of the few system settings we have, I might treat them as a new A/B cycle and recreate the complete system with the changed settings on the other partition. That way I also get automated fallback if anything is misconfigured and (for example) the device fails to connect to the new WiFi.

Does this compare or relate to Ubuntu Snaps? Obviously not the RO portion, but the handling the updates et al?

Otherwise this is really good to know. I guess I don't run RPi's enough because my lazy unplugs haven't been an issue...yet.

It's not based on Ubuntu Snaps. It's a custom development. It predated Ubuntu Snaps by I think 2 years? I don't see any obvious reason to switch to Snaps now, since it's all been really reliable so far. The downloads are done using zsync. Usually a system update is less than 5MB total.

This is pretty neat. Have you released anything used to build and manage the Pi aspect of this?

I haven't. So far it's really customized for our use case. If you're interested in a smallish self contained code that is based on Go, you might have a look at https://github.com/gokrazy/gokrazy

My old employer used the same A/B scheme for upgrades, and it did indeed work great.

Reply on news.ycombinator.com