Rolling your own minimal embedded Linux for the Raspberry Pi | Hacker News Reader

Rolling your own minimal embedded Linux for the Raspberry Pi

kevinboone.me

161 pointssnwy60 comments

This is a great series of tutorials to follow if you want to gain a better understanding of what goes into a basic Linux system. I think some of the commenters here are missing the point that it’s a learning exercise.

For production-grade custom Linux firmware, two of the most popular options are Buildroot and OpenEmbedded (Yocto). I’d recommend that anyone interested in this space get started with Buildroot. It’s really easy to make a basic image and customize it to your needs, plus you’ll learn a lot along the way. Yocto is much more powerful and customizable, but the learning curve is far more difficult. It’s great for more advanced situations where you need to support multiple boards or bring in 3rd-party layers.

I second that, Buildroot is great

PTXdist is also pretty good and very similar to Buildroot. My experience is a bit dated but PTXdist had better support for some boards than Buildroot.

I have built several embedded linux distributions for raspberry pi using buildroot. I have had great luck with it. There is a bit of a learning curve, but that seems to be the case with most embedded devices. I would recommend anyone getting started to try it out.

My favorite is using Nerves system that uses buildroot and boots directly to Elixir. Buildroot can be a bit of a pain but it’s fun to get a 20 mb Linux image with a few commands.

I would narrow that recommendation to just BuildRoot TBH. It fits much more with the kernel-config style whereas Yocto config seems like its from another planet.

BuildRoot is much easier to work with IMO and gives equivalent or better results.

Don't know when's the last time you've played with Yocto, but at this point all you need is to learn how to use devtool and that automates everything for you.

A couple of suggestions for minimal systems for the Raspberry Pi:

https://buildroot.org build the whole Linux system from source

https://gokrazy.org pure-Go userland for your Raspberry Pi 3/4 appliances

Don't forget Yocto(guide here: https://medium.com/nerd-for-tech/build-your-own-linux-image-...).

I think this should be the real tutorial here. The amount of flexibility that Yocto brings is huge compared to anything else you can do from any other distribution.

what's the usage of memory/storage comparing to buildroot for similar functionalities? I would expect something like 10x larger but I never measured it.

Sorry, I did not understand your question. What are the two things being compared?

Maybe buildroot and gocrazy?

right, golang seems quite 'fat' for many embedded use cases.

Maybe. But golang is probably okay for the kinds of things people run on a Raspberry Pi.

There is also TinyGo for microcontrollers. https://tinygo.org

Maybe it's just me, but I get forwarded to an advert for iFastNet and the domain gets forwarded to suspended-website.com.

I got that problem too.

I suspect the flood of visitors from HN exceeded the hosting account's quota.

https://web.archive.org/web/20210901081906/https://kevinboon...

It claims there's no supported way to boot from a read-only filesystem with Raspberry Pi OS, but there is. Setting it up is built into raspi-config, at:

Performance Options > Overlay File System

Rolling your own minimal embedded Linux is all cute fun. But what about maintaining it ?

Assuming you're not going to be rebuilding the whole lot manually each time, you're also going to have to go through the rather cute experience of building (and maintaining) your own automation tooling around maintaining your minimal embedded Linux.

If I were developing an embedded project, I don't know about you, but I would suggest focusing on the project itself might be a nice idea, not wasting time on things like a custom OS.

Custom OS is fine if you've got a large team working along side you. But if its a one-man-band (or small team) then all the cuteness will soon evaporate.

If you think all Linux distros are too bloated, why not go BSD instead ?

For example, OpenBSD is famous for a minimal default install, and guess what, they maintain ready-to-go builds for Pine 64/64+, Raspberry Pi 3 and Raspberry Pi 4.[1]

[1] https://www.openbsd.org/arm64.html

The Raspberry Pi is a learning platform, first and foremost. This is actually a great learning project for someone to understand what goes into a Linux system.

> If I were developing an embedded project, I don't know about you, but I would suggest focusing on the project itself might be a nice idea, not wasting time on things like a custom OS

Using an off the shelf distribution is ideal your goal is rapid prototyping, but most real-world embedded projects end up requiring custom distributions sooner or later. Distributions tailored to interactive use tend not to be so great for distributing and upgrading predictable firmware. You really want all of your devices in a known state, not a random mix of package versions depending on when they last ran ‘apt-get upgrade’, for example.

In production-grade products, the OS is often built with tools like Buildroot or OpenEmbedded (Yocto), not from scratch. However, doing a tutorial like this once is a good learning exercise.

RPI also sells modules for commercial products, so anything you play with RPI can be used on products too.

I've asked people how to easily download the sources for any given Linux distro to compile locally, and I've gotten links to sites with pages of instructions.

To build NetBSD/aarch64 on most Unix / Linux OSes (replace -j 8 with a suitable number for however many processor cores / threads you have):

  curl "https://cdn.netbsd.org/pub/NetBSD/NetBSD-current/tar_files/src.tar.gz" -o src.tar.gz
  curl "https://cdn.netbsd.org/pub/NetBSD/NetBSD-current/tar_files/xsrc.tar.gz" -o xsrc.tar.gz
  tar xzf src.tar.gz ; tar xzf xsrc.tar.gz
  cd src
  ./build.sh -m evbarm -a aarch64 -x -U -j 8 -D ../dest -O ../obj -T ../tools -R ../sets tools release

Is there really nothing this simple for Linux distros?

Maybe nix/guix, but most distros are going to have difficulty because they aren't a single thing to build, they're a collection of separate packages that you'd have to build individually.

Cannot offer any advice to the parent^1 however the build.sh instructions he provides bring to mind a general observation: By comparison, running buildroot's script will initiate downloads of files. This is a key distinction I see with Linux/GNU distributions (cf. the Linux kernel) in general (there are exceptions). Linux distribution maintainers try to automate everything. The idea of the user downloading the source tree for the entire distribution manually is not contemplated. Another example: When NetBSD is first installed, generally no networking programs are enabled by default. Its expected the user will turn things on, if she wants them on. By contrast, when a popular Linux distribution is installed, there are usually networking programs enabled by default to listen on the network. I use both Linux and NetBSD. I am not invoking a Linux v BSD debate nor arguing for one's approach over the other's, I am just pointing out how they are philosophically different.

1 The "NetBSD way" would be to study the NetBSD source tree and then modify certain files to suit one's needs. For example, I like to create custom crunched binaries; this requires editing a list. By comparsion, with Linux, I have never read any advice to study the Debian source tree and then modify files to suit one's needs. Rather, the advice one finds directs readers to Yocto, buildroot, etc.

I understand and appreciate that. But the distinction between manually downloading or automatically downloading isn't important to me.

Perhaps buildroot does the closest thing. I'll have to give it a go.

Maybe Gentoo is the closest thing.

You should really try out Yocto. There is a lot to learn but you can configure it from super-minimal tiny distros to mega kitchen-sink-included distros.

Everything is customizable, every file, every compiler flag and option of every package, with easy overrides in your own layer which only 'appends' to the original instructions, or adding and removing packages as you see fit in the base install.

What I dont underatand (yet) is in the example you provided you gave "qemuarm64" as the BSP. But the NetBSD example johnklos gave is not just for QEMU. I dont see an option for the RPi in poky's local.conf. NetBSD of already has configs for bootable RPi in its tree. This illustrates my point about automated downloads. Yocto requires, among other things, wget. For building images. This implies there is more to be downloaded before we can begin. With NetBSD, network access is not required to build images.

Uh, with Yocto at least, it's also pretty simple for a basic bootable disk image/packages... downloading all the source and compiling the cross toolchain along the way...

    git clone git://git.yoctoproject.org/poky
    cd poky
    git checkout -b hardknott -t origin/hardknott
    source oe-init-build-env
    MACHINE=qemuarm64 bitbake core-image-sato

But after the first build you may want more - bringing in other functional layers for different features (maybe you want docker?), support for different boards (same OS distro for x86_64 and ARM, or your own custom board... plus it can build a whole SDK for you for local development/debugging?), customization of target files, pulling from updated upstreams, support in deployment and throughout the device lifecycle, custom package feeds, etc.

There's a lot going for Yocto, it has a deep learning curve but does a pretty good job of taming the complexity of managing an entire OS distro.

I would use buildroot for that. It's very convenient even to just build random kernels for random targets

esxi for arm edition runs on the rpi 4 and 8 gb models. This way you can use the vsphere tooling and api to make it pretty painless, including the use of snapshots and cloning

Wow! Had no idea that VMware would run on a rpi. Thanks.

It’s interesting to think about the different markets for the rpi. I would never consider the rpi for learning. I only look at it as an inexpensive system to run my favorite distro. I want it to run CentOS or NetBSD, etc. rpi OS is meaningless to me. I want it to run the stuff I use every day.

>If I were developing an embedded project, I don't know about you, but I would suggest focusing on the project itself might be a nice idea, not wasting time on things like a custom OS.

And that's why you're probably not working on embedded. Minimal environments with almost zero footprint is a must in almost all cases. If you're rolling a custom board, you simply have to go custom, which is where OpenBSD will fall short on you. Check Variscite's or Toradex websites and see if you find any BSD in there.

The other thing is that existing distros are rarely good for anything truly autonomous. Too many moving parts. There's always the expectation that someone with a display and a keyboard, or a sysadmin at the console, is ready to pick up the pieces when it falls apart.

This is interesting. I have always worked with baremetal systems. Are there any good courses for understanding modern Embedded Linux standards who for shops who end up using a COTS dev board?

Rolling your own Embedded Linux on a device like the Pi is all fun and games for hobbyists until you run into the most typical security vulnerabilities and maintenance hell which comes from being unable to update it to the latest firmware.

Maybe we will see improvements in this space once other secure alternative operating systems and kernels designed for running on embedded systems are available.

Not on Linux distribution but this Arm sponsored textbook 'Operating Systems Foundations with Linux on the Raspberry Pi' is a useful learning guide:

https://www.arm.com/resources/education/books/operating-syst...

Looks good, sponsored usually you get it for free though.

typo in early commands of the form mkdir -p staging boot (space should be /)

also, busybox doesn't demand symlinks: you can use hardlinks.

If I wanted to build a RPi based system that has good support for bluetooth, audio (ALSA) and wifi, with fast boot times, what would be the most sensible route? Alpine or Buildroot? What are the advantages and drawbacks of them?

Alternatively, if you're not stuck on Linux, just install OpenBSD or NetBSD. Both will try to install X by default, but you can easily disable this in the TUI installers. After a reboot you'll go straight to a shell with very few daemons running (and certainly no systemd). Install the packages you need and start the daemons you need from there.

Imagine having the entire output of `top` fit in a terminal window with empty space at the bottom.

I do just this with Alpine Linux. I've been running it on Raspberry Pi Zeros with great success, and less than 18MB read only ramdisk.

I really wanted to like Alpine, but I can't for the life of me understand why it wants to do a ramdisk instead of just plain old read-only filesystem?

RAM disk is way faster and more reliable than SD cards. You can even remove the SD cards after boot.

Everything that's used should be cached in RAM, anyway. Everything that isn't used, well, I prefer that not to be in RAM. These things don't exactly have gobs of it.

And why would I want to remove the SD card after boot? Now I can't reboot the system, and it won't reboot if it crashes?

> Everything that's used should be cached in RAM

This isn't guaranteed, depending on many factors.

> why would I want to remove the SD card after boot

It's unnecessary to remove it. It's more about reliability. SD cards are notoriously unreliable, and it could potentially fail after boot. Coupled with the previous point, it has happened multiple times that I wanted to login to troubleshoot the system, but cannot because the basic system binaries like `ls` are not cached in RAM (as you said these are not used for the normal operation of these unattended systems).

This is when RAM disk is very helpful: as long as the system is powered on, I have reasonable confidence that I can login and see what's going wrong. It's mostly because RAM is way more reliable, faster, and cheap enough to "waste" a few dozens megabytes for the entire base OS.

Also there're diskless PXE systems where you just load the OS from network only once upon boot.

Alternatively, you can use Void Linux and most of the work is done... admittedly, no read only fs by default.

Link to kevinboone.me now redirects to suspended-website.com which then cycles through crap links.

Does anyone who's read this want to weigh in on how it holds up against Linux From Scratch?

This is really interesting! Thank you so much!

this is a remarkably well designed website, thanks for sharing.

omg

link is not work, domain is for sale...

Systemd and dbus are standard parts of the Linux userland now, nearly as essential as the kernel itself as they provide vital services.

Plenty embedded projects developed today don't use them (and functional desktop and server distros without them are around), "nearly as essential as the kernel" is really overstating it.

We'll be headed down this path soon for some custom appliances. I can't find a compelling reason for systemd or dbus.

The fact that desktop Linux systems mostly run systemd is (unfortunately) a reason to run a Linux distro that also runs systemd on new embedded projects (for large values of embedded. Storage is cheap these days, as is a CPU with an MMU).

If you can fit it in your image, systemd is amazing for embedded. Dependency tracking, auto service restarts on crash, unified start/stop and enable/disable, even journald is nice with per-unit unified logging.

It is miles better than SysV or monit. I haven't needed to try the 'suggested' replacements like runit or openrc.

Which are all great festures for devices where you'll have interaction. Auto restart on failure has been done for years, so the weight of systemd doesn't justify it for us. We'll have a single binary service, so dependency tracking isn't really a problem. Same for journald. 100% of the useful logs will be moved off the device, so the system journal isn't useful.

Not every nail needs to be hut by the systemd hammer. We've had really fantastic and light weight init systems for decades without all the bloat and creep of systemd.

Reply on news.ycombinator.com