Debian vs. Yocto for Embedded Systems | Hacker News Reader

Debian vs. Yocto for Embedded Systems

bytesnap.com

22 pointsteleforce47 comments

This comparison doesn't make a whole lot of sense to me. It's like saying "when I need a dessert, I prefer pie to gelato". I mean, yes, they're both dessert, but they're not the same category of thing.

Debian is great, and it's somewhat famous for the wide varieties of architectures it runs on. But fundamentally, it's an OS meant for use on systems that look a lot more like a computer than an embedded device. It's true, there's a good amount of overlap these days -- the most common OS for Raspberry Pi's is a Debian derivative, Raspbian -- but if you were building an IP camera, it's hardly the first thing I'd recommend.

Yocto, on the other hand, isn't really a distribution of Linux as much as it is a pile of code that lets you craft your own distribution. In a way, it shares some attributes with Gentoo here, though much more narrowly focused. Yocto is meant to produce images that get "flashed", even if not literally to EEPROM; it has integrations with tools like mender, that allow A/B/x style failover on updates. You'd expect things made with Yocto to be upgraded in a single block -- OS, code, all of it -- because it's all meant to run together. Updating a single package isn't really how Yocto is meant to work.

All in all, I mean... yes, I agree with the fundamental point: in most circumstances, when working on low-resource embedded devices, you're probably going to want something more like Yocto than like Debian, but, also, it's a very strange comparison to make.

I don't think it's a that strange comparison when you start looking at the end customers. There's certainly a market, mainly small to medium scale industrial automation, that really wants to write custom software to control machinery, but doesn't have the necessary background to run a meta-distribution system like Yocto.

Most SoC vendors provide a Debian image precisely for this market segment. The thing is, a lot of the guys who need a Linux embedded system are not embedded engineers, so they greatly benefit from having something like Debian, which has infinite resources to build upon.

Any vendor supplied distribution will be abandon and become horribly out of date. Maintaining it would be much harder than a similar Yocto system. They would better off embracing their reality than denying it: embedded engineering requires embedded engineers.

Source: paid to work on bringing up and maintaining Yocto systems every day.

I agree from an engineering standpoint but disagree from a market one. It would be a massive overhead for the market segment I mentioned before if they always went with "the best practices".

Source: same as you.

Unfortunately a lot of engineers who do this don't appreciate that flash has limited write cycles, and the issues with upgrading non-reproducible systems down the line.

A lot of projects using Yocto are not even reproducible after a couple years because people forget to save `downloads` and `sstate-cache`, I might add.

That's a good point. You don't need sstate-cache, but you do need downloads/ if you want to protect yourself from 3rd party sources going offline. I'd recommend archiving that folder alongside any released versions.

One thing that Yocto (and buildroot) do really well is reproducibility.

When building Debian/Ubuntu based firmware, you don't really know what package versions you'll get unless you are mirroring the package tree yourself.

If you don't, then any build-flow that involves apt-upgrading or apt-installing packages (yes, even Docker builds) has no guarantee that you'll get an equivalent result if you rerun it 3 months later.

That's funny because I'm working on a Yocto project and I'm thinking of porting it to Nix.

If you do, I'd love to hear a follow-up on how it goes. I keep meaning to try that out myself.

If you remember later, shoot me an email at nicholas-replace-with-dot-clark at gmaildotcom

If you mirror and snapshot the package repo you can get reproducible-ish builds. We have done this at very large scale at $job, see my post here: https://blog.heckel.io/2019/09/18/image-based-upgrades-upgra...

Debian stable also aims for equivalent results. But you are right that they might not be the same under a particular definition of reproducibility.

In general, once you have a build environment that is actively maintain (to fix important bugs and security vulnerabilities), it will approach Debian stable to some degree, in the sense that developers get updates they did not request (so that they stop baking in known and fixed vulnerabilities into their applications).

> any build-flow that involves apt-upgrading or apt-installing packages (yes, even Docker builds) has no guarantee that you'll get an equivalent result if you rerun it 3 months later.

You can ask `apt` to install specific versions of packages. You do have to provide a repository that contains old versions though, e.g. Debian's snapshot archive: https://snapshot.debian.org/

Yocto has no place on any machine that isn't heavily resource constrained. A "normal" OS like Debian gives you multitudes of advantages at the cost of a little extra disk space: thousands of packages available with one command, tons of available tools, support for many different architectures, cross-building. AND you can do both at the same time. At $JOB we have hardware with a vendor-provided yocto thing that has some binary hardware drivers. So we can't dump yocto completely. Instead we have Debian running in a chroot, talking to the binary driver running in native yocto via a pipe. Recommended!

I've found yocto package availability quite good, although you may need to add some non-default meta layers. Do you have some examples of what packages and especially architectures Yocto is missing compared to debian?

Also I would be interested to hear how debian has advantages in cross-building; I thought they native-compile all of their packages. Is that really the case that in debian that you can modify some existing deb source package (eg. change configure flags or apply some patch) and cross compile that into a .deb, like you can in Yocto?

Debian is a mainstream OS used by millions, while yocto is niche. Package availability isn't comparable, and you don't have to look hard to find missing things.

And yes. I can install a cross-compiler (from the stock repo!) and custom-build whatever (not 100% of the archive is cross-buildable yet, but most of it is). And most of the time you don't care about the details, and you don't need to cross-build anything: the full Debian archive is pre-built and available on all the supported architectures (whatever you care about is probably on that list).

And there's another big advantage. Developers might be using a different-architecture machine for their testing, and it's really nice to have an identical set of installed packages and version across dev and deployment boxes, even if they have different architectures. This is actually a very common case: people develop on their amd64 laptops, but the deployment might happen on an arm, or something.

That's a solid approach. Just make sure $JOB's legal dept is OK with Debian's license requirements (you have to tell customers how to modify your chroot), and with potential traceability/software supply-chain concerns.

Yocto does have other benefits than just resource usage. It's also good if you want to customize your compiler options, or if you operate in a safety-critical context where you might need to trace the origins of your compiled binaries.

You can custom-build Debian packages in the same way. Rarely does anyone actually do it because it rarely actually matters.

Cool, here's something I can talk about!

At Toradex we provide an operating system and OTA platform called Torizon which is Yocto-based but uses Debian containers with custom packages for hardware acceleration, so our customers have the customisation of Yocto (if they so desire) while getting a very familiar programming environment (Debian). "The best of both worlds", so to say.

Customers are very happy to not deal with the usual Yocto mess.

[1] https://developer.toradex.com/torizon/provided-containers/de...

Disclosure: I'm a software engineer at Toradex.

what's the hardware requirement for this to work efficiently(e.g. cpu freq, how many cores, memory,storage)? contains are sometimes heavy for many embedded boards where RAM and storage and power consumption are typically limited.

The processing overhead is pretty minimal (take a look at our iMX6ULL, for example), because the containers are just fancy processes at the end of the day. The only real drawback we see is boot time, but you can optimise that as well.

container could be really heavy to me, as it shares nothing with the host(other than the kernel some volume mounting), lxc seems much light-weight to me as it shares the host libraries and packages.

Openwrt also makes an excellent base embedded distro - it's designed to run from a read-only root which is important for any serious embedded device, whilst still having the ability to add packages. The common ringlogger simplifies system logging, and the UCI settings stuff is well thought out and simplifies device configuration backups/deployment.

While OpenWRT is really good and often the best one for really constrained Linux HW, adding features like secure-boot and secure storage to the builds can be much more difficult than on Yocto. Especially when vendors like NXP provide that functionality in their Yocto BSPs. This can be important for commercial projects because a lot of customers are now asking for such features to secure end-devices.

Yes, excellent point regarding secure boot (for good or for evil)!

That's true, but converting recipes to Openwrt is not that hard, it's just tedious.

Please do not forget Buildroot for embedded Linux. You end up rebuilding all, but its much simpler than yocto, and based on makefiles.

Buildroot did an excellent job of converting me into a lifelong yocto advocate. Yes it's much simpler upfront, but the sheer number of subtly broken builds will slowly drive you insane, since you're not going to be patient enough to wait 30-60m for full rebuilds every time you change something.

Most vendors are on yocto nowadays anyway. If you stick with buildroot you'll be doing all the work yourself.

I second this. I worked with yocto for 8 years and my current job uses buildroot on all their projects and I'm trying to get them to convert. I'm supposed to give a company wide yocto training in october. Gonna do my best to sell it well.

buildroot is good for board bringup to verify if the hardware design works, much more simple and quick than yocto, after that though, yocto might have advantages to make real products.

I love Yocto for embedded work, and used several distributions before that used Yocto at the core like Wind River Linux, Ostro, or BalenaOS, as most of the times embeeded or IoT/IIoT lacks in the resources department, and mostly are using arm-based SBCs like Rpi or Jetson Nano, making it a no brainer for such cases. For the users who are questioning the package management or support issues, a way is to use containers and run base image, getting the best of both worlds!

Most SoC vendors will give you a Yocto that you will use for bringup/hardware validation. So you're stuck with it at least in the beginning.

And if your device has limited storage...

Another nice alternative is pmbootstrap, postmarketOS's one-stop tool to setup cross compilation environments, cross-compile packages, run tests in qemu, flash devices, etc.

I know some projects picked it up and adapted it to their purposes, it's relatively simple and written in Python.

https://wiki.postmarketos.org/wiki/Pmbootstrap

I don't know anything about Total Annihilation, but all the versions since dunfell are also locations in the English Lake District. At a stretch you could include Dunfell too if you allow locations in wider Cumbria.

Debian has documentation.

We have use debian for years. Our controllers are more equipped than a RPi. We also do our own APT repos in docker, and can create custom APT repos for any client for any project. Our application developers love using APT for development as they can surgically update working images for testing. We have our own mirrors to ensure that things are consistent and rebuildable and use docker for building images from debbootstrap which also have the advantage of making quick test images. The other advantage is that debian is a 'force multiplier', everything we need is there and it allows our devs to go fast. Yocto is fine and is very standard, but we made it work here really well.

But the (yocto) documentation could use some work.

Debian has hard dependencies on a ton of GPLv3 packages (coreutils, bash, etc). As a result, you can't legally ship a product containing Debian unless you give your users the ability to modify your root filesystem.

So: if you're making a consumer-facing product that has secured firmware updates, then Debian (or any derivative like Ubuntu) is probably not what you want. Assuming you care about license compliance.

This interpretation of GPL compliance doesn’t match anything I’ve seen before.

Have you ever read GPLv3? It has a so-called anti-tivoization clause that covers exactly what I'm describing.

That clause was the reason why the Linux kernel stayed on GPLv2, and so did a lot of core embedded software like BusyBox.

That's not quite true. First of all, the anti-tivoization clause only applies to things "normally used for personal, family, or household purposes" or things "sold for incorporation into a dwelling". You can run GPLv3 software on your MRI machine as much as you want, without having to let anyone else install their own version of it.

Second of all, even for consumer products, you can still make it so that you no longer trust machines that have been tampered with -- as long as you still let the person execute their own code. For example, Chromebooks getting put into developer mode by removing a screw could be cut off from cloud resources, or a phone screaming at you on boot that it's no longer trustworthy.

Good point about commercial/industrial devices.

For consumer products, we are both right. A consumer product developer must provide owners with any secrets/keys/code necessary to replace any of the GPLv3 software. So: rights to modify the rootfs.

It's OK for the product to complain at you for doing it, but the vendor has to tell you how, if you ask. And give you any signing keys. That's not ideal on products that might have software-unlockable features, or on public-facing things like routers.

The second is true, but it is also letting your customers have root (as the OP said.)

It's control through the threat of ending a relationship, which Redhat is also trying now. You can't really say that it's allowed for in the GPL (there's no text in the GPL that says that it is allowed), it's just something that the GPL does not cover.

And third of all, the reason why Linux and busybox are not in GPLv3 is because it is to complicated too change the license.

The anti-tivoization clause does not require root filesystem access - that is one path allowing modification, but not a particularly secure one. keeping a secured booth path to unmodified original firmware&fs is fine as long as access to an alternate bootpath of the device in order to use modified software is allowed.

This sounds like a feature, not a bug.

Reply on news.ycombinator.com