Building Raspberry Pi Systems with Yocto | Hacker News Reader

Building Raspberry Pi Systems with Yocto

jumpnowtek.com

80 pointscodetrotter30 comments

Just a quick tip, if you are interested in Yocto, and especially Yocto on the Raspberry Pi: there's a layer called meta-updater[1], originally developed for Automotive Grade Linux, that lets you add over-the-air updates to your Yocto-built systems, and it supports Raspberry Pi. It uses OSTree[2] to give you atomic updates and nice, small update sizes. (The whole filesystem is a content-addressed object store that's hard-linked into the actual directory structure at boot time.)

There's also a guide[3] for doing this using HERE's servers for delivering your updates. It's free to use, but if you prefer a free-as-in-speech solution, you can either ignore the update client and run the updates yourself from a bare OSTree repo, or run the open-source community edition[4] of HERE's server software.

(Full disclosure: I work for HERE, and wrote the quickstart guide[3] I'm linking.)

[1] https://github.com/advancedtelematic/meta-updater/

[2] https://ostree.readthedocs.io/en/latest/

[3] https://docs.atsgarage.com/quickstarts/raspberry-pi.html

[4] https://github.com/advancedtelematic/ota-community-edition/

With regards to image update management, personally what I wish to do for my Yocto image is to have it be:

- Small enough that it fits into RAM.

- Fetched over the network on boot, PXE-style.

- Configured to not write any files to the greatest extent possible, and where writing files can not be avoided one would use tmpfs.

With the above three criteria satisfied, the following is achieved:

a) We never write to an SD card so we avoid SD card corruption issues.

b) Since the whole system image is sitting in RAM, we could do without NFS, Samba or similar either.

We do need to persist some data, but that data is coming from our application and is going to be stored in a database on a server on the local network.

I am currently working on developing the application that our image will run, but I have dug up some preliminary information relating to what I said above:

- Read-only Raspberry Pi [1]. Applies to Raspbian but certainly provides a starting point for doing the same thing with Yocto.

- Poky NFS Root [2]. Like I said, I don't need the NFS root part if the image can fit in RAM, but the portions that deal with gPXE might certainly be relevant.

- The State of Netbooting Raspberry Pis (December 2017) [3]. "Net-booting works well on the Raspberry Pi B Model 3 without an SD Card." So actually we won't be needing gPXE.

- Network Boot Your Raspberry Pi [4]. Referenced by [3] and provides the steps needed to configure Raspberry Pi 3 for network booting without an SD-card.

I am not completely decided upon whether or not to avoid NFS yet though. Because we never persist any files on the root file system anyway, serving said fs over NFS can be done in read-only mode, and we would put each revision of the corresponding root file system in a uniquely named top-level directory so that a client running the current version image has one place that it gets the files it expects from and the next version of the image which might expect other files or differing layout would get its files from another directory. Old root file system directories which are no longer in use would be removed so this scheme doesn't require much in terms of storage space.

It should be noted that NFS has some security issues [5], but our deployment is going to be running on an isolated network exclusive to the hardware that we are providing, so this concern is already resolved. Said isolated network will connect our embedded Raspberry Pis, a server (TFTP, database and NFS), and a tablet configured and provided by us. The tablet provides a graphical user interface that our customers will use to interact with the system.

[1]: https://learn.adafruit.com/read-only-raspberry-pi/

[2]: https://wiki.yoctoproject.org/wiki/Poky_NFS_Root

[3]: https://blog.alexellis.io/the-state-of-netbooting-raspberry-...

[4]: https://www.raspberrypi.org/documentation/hardware/raspberry...

[5]: https://www.tldp.org/HOWTO/NFS-HOWTO/security.html

Running an OS + application entirely from memory is an interesting approach indeed.

I guess the fundamental big problem you are facing is SD card corruption. In our experience, we've found the SanDisk Extreme Pro cards to be most reliable. (coupled with a good power supply).

Read-only rootfs is almost a must have for any production environment.

resinOS does not satisfy all your requirements (nfs boot, state in memory etc).. But we do have stuff in place to reduce the problems you are facing.

- There is an initramfs in the kernel. When the kernel loads in memory, it runs fsck on the root/data partitions before mounting the rootfs. - The rootfs is read-only with only a few configuration files in the state partition. - Applications run inside a container so you can basically run rasbian on top of resinOS.

I wrote more about resinOS in another comment highlighting remote application container updates/vpn access etc.

https://news.ycombinator.com/item?id=18093336

Disclaimer: I work at resin in the OS team. I've noted down your feedback about running purely in memory.

> Applications run inside a container so you can basically run rasbian on top of resinOS.

How well does that work? Do you get full access to the underlying hardware? Can you still access the GPIOs, HDMI-CEC etc? Are there any other downsides to doing this?

Sorry for the delay. Yes they get full access. CEC might be an interesting one. "If you choose, your containers can be configured to run as privileged, access hardware directly, and even inject modules into the kernel." from https://docs.resin.io/learn/welcome/primer/

Be wary of some packages that expect their log folder to exist at /var/log/some_package else they won't start, I can't remember which of nginix or apache has that issue. That adafruit script is very simple and there are more robust setups out there, albeit not as well documented.

Netbooting isn't 100% reliable and last time I checked there were certain network switches/hubs that caused issues. Maybe that's a non-issue if you are also providing the network infrastructure.

Lastly, remember that the more RAM you consume for your filesystem/logs etc, the less available for our actual application. That becomes more of a problem when you start also assigning a large chunk of that RAM to the GPU.

> Netbooting isn't 100% reliable and last time I checked there were certain network switches/hubs that caused issues. Maybe that's a non-issue if you are also providing the network infrastructure.

We will be providing the network infrastructure indeed, but being aware of this will certainly be valuable if we face those sorts of problems, so thanks for that :)

Do you happen to know of any specific switches or hubs that would work reliably in this context?

I'm sorry I don't remember what I have but [1] suggests a Netgear GS108. That link also explains a couple of the issues and they may have even been fixed in more recent bootcode versions. The 3B+ has a later version (and also has netboot enabled by default so you don't need to mess with program_usb_boot_mode).

[1] https://www.raspberrypi.org/blog/pi-3-booting-part-ii-ethern...

The Yocto package format supports a mix of inheriting BitBake package files, Bash and Python. It's over-engineered and unpleasant. Values can easily be defined 3 or 4 layers away from the file you are looking at, either in one of the many configuration files (that can be overridden by other configuration files), or in one of the files the package is inheriting from.

Add clunky and unintuitive tools to the mix, and Yocto can easily be an unpleasant experience, at least for teams over a certain size.

It does make switching between architectures really easy and sweet, though, but that's about it.

I'm currently using Yocto for an embedded Linux project, and agree that it just feels so overly complicated for what we're using it for. I've written packages for Arch Linux in the past, and that's a breeze compared to writing bitbake recipes.

The ability to modify existing recipes through append files is nice, but that seems like it could be handled in a normal distro just by forking a package and making changes.

i recommend anybody who shares this opinion should take a look at buildroot as an alternative. if you've ever configured and built the linux kernel, you'll feel right at home with buildroot's approach to configuring and building an entire filesystem image.

Having just found Yocto yesterday and built my first custom image that I made by extending the layer in the OP, I am not yet "married to" Yocto.

Yocto initially appeared to be exactly what I was looking for. When someone else raised concerns about Yocto ITT I somewhat brushed them off, mainly because they were talking about managing the layers on a headless system. I used Yocto from the command-line on my workstation, so a) I am not on a headless system and b) even if I was doing this on a headless server, everything I have done so far would be done exactly the same way.

But, since then, multiple other people have raised concerns about Yocto. I also talked to one of the mods on /r/raspberry_pi and they said something similar. I looked through some past discussions about Yocto on /r/raspberry_pi using the search feature on Reddit, and found a couple of other similar experiences.

After looking through what has been said, I find that these concerns resonate with me rather deeply. They are all pointing to a single sentiment that seems to be rooted in similar real-world experiences; Yocto makes things complicated in a way that with time will make managing things needlessly complex and which will result in a lot of wasted time.

I was still a bit inclined to think though that perhaps these experiences were had long enough ago that Yocto has since improved, but after a little bit of further thought I realized that fundamentally Yocto is engineered in the fashion that you and others are pointing to as being a fundamental flaw of the system.

I am going to try Buildroot instead. Same guy that wrote the post I shared in the OP also made a post about using Buildroot. https://jumpnowtek.com/rpi/Raspberry-Pi-Systems-with-Buildro...

Although Yocto gets more publicity, I always prefer BuildRoot myself.

Yocto's 'layer' abstraction seems overly complicated and difficult to use (especially in a headless situation where you can't use a nice GUI tool to manage them).

BuildRoot is much more like the Kernel Configuration... Just choose your options from a hierarchical menu.

Much easier IMO.

(https://buildroot.org/)

Also, building a minimal system is much easier in BuildRoot than in Yocto... just deselect the options rather than trying to remove layers and the contents of layers.

I found creating a layer of my own that extended the one in the OP with a couple of things extra was “easy” (in terms of what I would expect from the HN crowd. For a beginner it would not be easy), but I will agree that removing things will probably be more complicated.

Here are a couple of photos of my Raspberry Pi 3 with said Yocto image running some graphical applications without X11 (nor Wayland):

gstreamer videotestsrc ! glimagesink https://i.imgur.com/dPQqgRX.jpg

tspress https://i.imgur.com/CLx1RrD.jpg

In the time since I wrote the above comment I've decided to give Buildroot a shot. See https://news.ycombinator.com/item?id=18093599

Glad you tried it with an open mind....

For background, I'm a softie with over twenty years experience with embedded, desktop and server systems from microcontrollers to multi-core DSPs so I've a fair amount of experience in this area.

The last time I tried Yocto was while building a mesh router for an IoT product. When building real products, removing all cruft and only having a minimal system is very important from many viewpoints and this is crucially very difficult in Yocto. I gave up and went the BuildRoot route.

I just find that Yocto has overcomplicated something that is essentially very simple. BuildRoot on the other hand works exactly how I expect and therefore I'm much more productive using it.

I just finished building my first BuildRoot image and here are my thoughts.

The Yocto build system is sneaky because when it is running it shows the total number of tasks and the number of tasks that have executed, so it feels a bit more comfortable in that way than BuildRoot.

In this respect, BuildRoot vs Yocto feels a bit like running a big compilation of ports on FreeBSD using make directly vs running the compilation with poudriere. The latter has a web user interface that lets you track the progress a little bit. The web interface of poudriere looks like so: https://blog.shatow.net/static/poudriere-31.png

However, like I said, Yocto is sneaky. Because while the above mentioned aspect might seem like a big plus, the end result of using BuildRoot was very impressive.

I based my BuildRoot config on https://jumpnowtek.com/rpi/Raspberry-Pi-Systems-with-Buildro... with some additions and removing some things I didn't need.

I noticed also that there was an option "remount root filesystem read-write during boot" that could be disabled. Having a read-only image is something I was planning on doing already, so I gave it a shot. And it works!

Thanks to BuildRoot, I have a lean 251 MB (lean in the context of what I am doing, I am aware that some Linux images for some routers etc are just a handful of megabytes) that can run with a read-only rootfs and which does everything that Yocto image I had did, and it took much less time to compile because like you said, it was easy to customize it to only have what I needed basically.

My workstation has 32 GB of RAM and an 8 core, 16 threads AMD Ryzen 7 1700 Prosessor. Building the Yocto image took in excess of 3 hours, close to 4 hours I think. Building the BuildRoot image to 80 minutes, because the BuildRoot image, unlike the Yocto image did not have a lot of extra cruft, exactly because like you said, with BuildRoot it is easy to remove what you don't need.

Now one thing I did notice because I had overlooked some of the configuration options for one of the packages was that changing the options of a package does not cause BuildRoot to regard it as necessary to rebuild the package. Apparently this is by design [1]. I think that is a tiny bit disappointing, but whatever. Rebuilding individual packages is quite fast anyway. And hopefully if I need to do full rebuilds of the whole thing then ccache, which I made sure to enable before running the first build, will shave some time off the build. But I don't even think full rebuilds will be necessary all that often.

All in all I am glad that you suggested I try BuildRoot, and I am glad that I listened. I will be sticking with BuildRoot. Thank you :)

[1]: https://github.com/buildroot/buildroot/blob/master/docs/manu...

I looked into switching to yocto a while back for a rpi project I work on. However, one thing that put me off is the end user's inability to install regular debian (raspbian) packages as they will inevitably expect to. But I think that people have used yocto as a kind of docker hypervisor, the openhab project used to do something like that before moving to HassOS. Does anyone know if you can include a minimal raspbian-based docker image in your yocto system which users could then apt-get install whatever extra packages they want? Does that make sense?

Not tried this myself but this is probably what you are looking for

http://layers.openembedded.org/layerindex/branch/master/laye...

Even without meta-debian, you can configure yocto to use the apt package management tools, and then you can point this at the relevant repositories. Read the manual for more information.

Thanks for that. I'm not sure I understand yet if meta-raspberrypi and meta-debian can be used together, I'd have thought they are both at the same layer and so couldn't be. And it seems to be based on Jessie.

I did read about how to add APT by specifying: CORE_IMAGE_EXTRA_INSTALL += "apt" but how on earth does it know what you've 'baked' into your recipe? Otherwise surely it'd basically install the system again when handling the user's package's dependencies. Wouldn't it? Perhaps https://community.nxp.com/thread/325384#comment-486697 is the answer, which sounds like a bit of work.

I also found this which looks kind of interesting: https://elinux.org/images/c/c6/Smart_r002.pdf and https://community.nxp.com/docs/DOC-328199.

I recently ran into that issue - I have a nice fine-tuned yocto image and now a customer wants to "log in via ssh and install a custom raspbian package". I was thinking of the docker solution as well.

"We have small and fine-tuned ghetto, but customers don't want that. What to do?"

I have similar problem: we have consumer device on top of imx6. I used iMX6 SoloLite evaluation board kit (1GB of memory) with Fedora for arm as development prototype. For me, installing a package is not a problem at all: "dnf install package". I planed to use Raspbian or Fedberry on actual device, but our management blindly switched to Yocto, without comparing of alternatives. To install package on Yocto, we need to create our own recipe in our own layer,then rebuild Yocto, fix bugs and repeat, and then deliver update. We spent man-year to achieve same result as we already had with Fedora. We seriously lag behind schedule because two developers (and two consultants) are working on developing of our own custom OS instead of working on our application. Moreover, instead of single system for development/testing and production (Fedora or Debian on developer workstation, in Docker for CI, on actual device) we now have mix of two systems. I created configuration script with about fifty options to be able to compile for host and for Yocto at same time (including workaround for number of bugs in Yocto). Enormous development time was sink into ground because of this schizophrenia.

ouch. That sucks.

You could run Fedora in a container on your Yocto base using resinOS.

I wrote more about this in another comment. https://news.ycombinator.com/item?id=18093336

I already have OS. In my case, I just use chroot script to access full Fedora distro with all development tools on Yocto. If I will be able to replace Yocto, I will replace it with major distro, like Debian, Ubuntu, CentOS, or Fedora, not by yet another toy distro with thousands of packages per maintainer.

I'd recommend using resinOS as the base. And then running your application inside a Raspbian container.

That will give you a read-only resinOS (based on Yocto. meta-resin layer). If you rpi is connected to the internet, you can use resin.io to remotely push application container updates and access the pi over a vpn.

https://docs.resin.io/learn/getting-started/raspberrypi3/nod...

Disclosure, I work at resin.io in the OS team on meta-resin.

The base resinOS is open source. And we are open-sourcing the rest of our stuff in due time as well.

https://github.com/cmhe/meta-readonly-rootfs-overlay

Users can apt-get all they want and it'll even be available next boot provided they shut down properly.

I'm not sure this layer has much to do with providing apt-get support but maybe I misunderstand.

Unrelated but when I saw the title I thought it would be about yoctopuce, which are usb environment sensors that are easy to script for in python or other languages. I used them to create a few weather station loggers using raspberry pis.

http://www.yoctopuce.com

Those seem like fun products; though the hardware-tinkerer in me is thinking they could be home-made via ESP8266 hardware very very easily.

Most likely, but they are a good solution for someone like me who is comfortable with software, but doesn’t really want to invest time in the electronic / hardware. They are a bit pricey though (but I guess it’s not a big market).

Reply on news.ycombinator.com