Yocto: Tools for building a custom embedded Linux distribution | Hacker News Reader

Yocto: Tools for building a custom embedded Linux distribution

yoctoproject.org

87 pointsbased253 comments

https://www.yoctoproject.org

"The Yocto Project (YP) is an open source collaboration project that helps developers create custom Linux-based systems regardless of the hardware architecture."

Press releases should really have a description of the project or at least a link

Most people in the industry know about the two big players - yocto and buildroot. It's for when you need a custom minimal Linux distro for your gadget (IoT, router, modem, SoC, etc.)

Thanks. We've changed the url from https://www.yoctoproject.org/pipermail/yocto-announce/2019-O... to that, since the project doesn't seem to have been posted to HN before.

Though there was a related thread from 2018: https://news.ycombinator.com/item?id=18083506

I personally haven’t been impressed with yocto: It’s about as intuitive as gentoo and reliable as buildroot.

Maybe I was using it wrong though. I think it’s a good idea.

What sorts of reliability problems have you had with Buildroot? And what do you use instead?

Second that question. If you don't do product lines with varying platforms, buildroot is much easier.

Honestly I was just being rhetorical.

Buildroot really isn’t bad but it’s not totally solid and it’s not hard to mess it up either by not thinking or not reading the documentation.

Changing the config and rebuilding without cleaning is the most obvious thing although the docs explicitly tell you not to do that so that doesn’t really count.

Certain compiler build settings can result in a failed build.

It’s really only meant to be used for making cross compilers anyway and the root image generation is a sometimes useful thing. Personally if I want a rootfs it’s easy enough to make one from scratch.

I work with Yocto in my day job. The learning curve is really steep, but I actually like it a lot.

Buildroot a great project with clear goals and good execution. Yocto is flexible to a fault, which makes it a lot more complex. But with that complexity comes a lot of power and fine-grained control over your builds.

I work with Yocto at my day job too. I hate it. I have 20 years of experience with Linux, and my favorite disto is Fedora. Fedora on arm is working much better than Yocto (except for DNF: 40,000 of available packages is too much for an embedded system, so at least 1GB of memory is required to run DNF).

Buildroot at work. Someone told me yocto is the future so I spent a week with it but it was a nightmare. Went back to buildroot.

Gentoo is quite intuitive by comparison.

>Fedora on arm is working much better than Yocto

I concur. I've tried Yocto, was appalled by it. Then I've taken Centos 8, and it was a breeze to build for it, safe for some minor troubles.

I would choose Centos/opensuse/fedora/custom distro over Yocto anytime.

Yah, extra bonus because you can upgrade the system with dnf upgrade/etc. Unlike yocto where its a PITA and requires an engineer to validate it works every-time you need a security patch.

Yocto is just another "The S in IOT stands for security" kind of problem.

Fedora on ARM definitely works better than Yocto out-of-the-box. It's also much larger, and you don't have much control over the core components. You don't have control over how the packages were compiled, and you'd have a hard time reproducing your exact build later (especially if your flow includes a dnf update somewhere).

Yocto is the wrong tool if you're looking for a good developer Linux. It's a great tool if you're looking for a way to make a highly customized Linux image. It provides good reproducibility and strict license mechanisms to make sure you don't run afoul of anybody's open-source license. It also provides you with tools to compile and include your own recipes, and easy ways to modify upstream recipes without needing to fork the Yocto repos.

Fedberry is about same size as Yocto. Not a problem to customize RPM based distribution at all.

Having used both for extensive periods of time I just need to say that gentoo is at least an order of magnitude more intuitive.

Yocto is, well, "acceptable" once you get the hang of it but I definitely get where the sentiment is coming from.

At one point I was trying to build a completely minimal system for an already embedded architecture and instead of building a toolchain, target kernel and essentially empty userland I got 4000 build tasks, it was downloading the entire compendium of all imaginable open source software packages out there and projected a finish time of some 3 hours.

At least buildroot isn't that stupid, although of course it is still stuck in "recursive Makefile" hell.

For what's it's worth if you use the public shared state cache you can do a full build of core-image-minimal in 5min. Also I'm not sure how Yocto is being stupid by generating 4000 tasks, if you target something with a GUI like core-image-sato you are easily going to get there because there really is that much software involved. Also remember task does not match 1:1 to recipe, you are looking at 5-6 tasks per recipe usually. Hob can help you explore the dependency graph if you really are not sure why the build got so complex.

Hey, just for kicks, I did it once more! Here is what they say about core-image-minimal:

> The minimal recipe provides an image with the least number of packages to be a bootable image

Here is what it is building, incomplete list, for install to the target rootfs:

- libx11, libxau, libxcb, libxcrypt, libxdmcp, libxrandr, libxrender, glib-2.0, "gnome-desktop-testing", gperf, openssl, perl, pixman, libsdl2, python3, sqlite3, various xorg-..

This isn't even the freaking desktop image! Also, no, sorry, no more Hob, that is now deprecated in favor of a django webinterface (Toaster).

If you are building a software image for an embedded device and you don't know every package that is going into it and why it is there, you need to stop. Look to the Android AOSP for what it takes - an in-depth understanding of the toolchain, strict choices for compilers and libc (none of this clang-experimental and your choice of uclibc, musl, glibc), stringent standards across the entire body of code you are compiling - no "uh this doesn't compile with PIE and hardening, just hack the CFLAGS". And there are ton more lessons there.

Yep. What I'm doing with buildroot is maybe a couple dozen packages. Fresh build, with toolchain and all, takes around 20 minutes, and I end up with a 10MB image that contains the kernel & userland, plus another image with bootloader. It didn't take long to get there, the learning curve was very shallow.

I wanted to give yocto a chance but after a week of getting nowhere I want to be with it, I turned back to buildroot.

I'm not going to convince you, but some of what is being built there are native packages likely for things like QEMU. You need to look at the actual manifest in the deploy folder to see what got installed. You will find that it is quite small, not even including exta kernel modules, maybe 10MB or so.

I have shipped products with both buildroot and Yocto and I have to say I have fully moved over to Yocto now. Adding things like OTA are trivial, and I can inject extra tasks for enforcing rules or extracting license meta-data. But being able to create real packages is huge when you have a larger system and just want to test the new version of an application. With buildroot I have to manually copy the files around. Same thing if I want to add gdbserver, you cannot create a package repo to pull from you have to rebuild the whole image.

Well you know the computer only does what you tell it to do.

I think you're right.

I worked on Yocto for two years, it is over-engineered _a_ _lot_ to say the least.

The BSP companies like it though, my theory is that, since Yocto is so complex(unnecessarily), only them can solved the problem for clients.

It's a worse case comparing to gentoo for embedded space. Just use buildroot to bring up your board, much simpler and faster with top class documentation.

> The BSP companies like it though, my theory is that, since Yocto is so complex(unnecessarily), only them can solved the problem for clients.

I also think Yocto is mainly an advantage for BSP vendors, but I think where the benefits might offset the complexity is when managing lots of client specific variations and adaptations for the same base BSP.

So instead of "share code and save money", like normal distro, they use "split code and make profit" model.

>I personally haven’t been impressed with yocto

That's a very mild way to express my feeling about it.

Maybe I'm asking too much, but Yocto/openembedded is the worst piece of software I've ever dealt with.

The build system is a huge pile of opaque and fragile scripts. It's so fragile, that if you don't put a space before a list of options, it would be concatenated with other options without a space, leading to incorrect options.

Because there are no lists, lists are just strings of space-separated words. Who needs proper data types?

There are no modularity either, everything is configured through strange variables with names like CONFIG_image_graphics_mesa_package=... God knows how this naming works.

Of course nothing is documented, and the discoverability is extremely low: everything is done through hundreds of opaque shell scripts, python scripts, half-baked python scripts for their build system.

The overall design of the build system is abysmally bad and fragile, and it's harder to get a notion how the things work than to just build everything with your own scripts.

Oh, and when something is wrong, build system just returns python stacktrace.

Is it that hard to use proper abstractions, data types, docstrings, introspection mechanisms so that everything would become transparent, obvious and sane?

Example: PACKAGE_native_append += "something". "append" is a keyword, but the "native" is just a separate list of native packages, yet both are appended to a variable name separating it by underscores. They use the same "syntax". If my memory serves me correctly, if not in this example "+=" is required sometimes after "append", even if they seem redundant. Someone politely explains you why this works as it does, it makes sense, yet you wonder why you need to hit a gotcha when all you are doing is something as simple as adding an item to a list (as a space separated string).

And then the metadata is also funny, e.g. do_patch[depends] = "whatever:do_populate...". Variables are not just variables, they can contain metadata, sometimes important.

It feels like there are usage patterns, good practices and something more beautiful wanting to get out, but this is a domain where making a better build tool would break tons of recipes, so it feels like everytime they got a problem they added a quickfix and got forward without looking back.

The steep learning curve most of the time is unneeded, you are frenquently doing trivial to simple taks and hit one problem that gets you blocked for a while for things that you are easily able to do outside Yocto.

It is a shame that the good work they do is spoiled by the horrible "interface".

I hated it, too. Also, OpenWRT, what a nightmare to build against. Now if netbsd was ported to the zipit z2, I would switch in no time.

> Of course nothing is documented, and

This is plain wrong. There is a whole bunch of manuals https://www.yoctoproject.org/docs/

And there are several books https://www.yoctoproject.org/learn/books/

Without understanding the philosophy behind the whole system and the recipe language just trying to "discover" something seems indeed mission impossible to me. Yes, the learning curve is steep. The recipe language is not always the least surprise if you don't understand it. On the other side, there is a lot of magic support in the system. If you know what you are doing writing recipes for new packages does not require much code.

> There is a whole bunch of manuals

Ah, yeah, the manual. I forgot another abysmally bad part of Yocto: the manual.

Yocto documentation is the single worst documentation I've ever dealt with. It's a huge giant wall of text which tells you... nothing concrete whatsoever.

Wanna know how to change an image fs or image format? Go figure with the manual, just look at these entries:

https://www.yoctoproject.org/docs/3.0/ref-manual/ref-manual....

So concise, but even more useless. Which filesystems could I use? How to know that? Ah, don't bother.

The whole manual is a huge wall of text telling you what classes, tasks, recipes and dozens of other overengineered concepts are, but nothing about how to use them beyond the basic examples. Wanna build an fs for the device with a local storage and external ssd, good luck to figure out how: decoupling where machine starts and image ends, and which part should you modify.

> Without understanding the philosophy behind the whole system and the recipe language just trying to "discover" something seems indeed mission impossible to me.

Yeah, it's hard to introspect something which have dozens of concepts, yet no other form of introspection but file reading.

And I'm fine with file reading, but with yocto half of important stuff is concealed within the build system, leaving just strange config variables and gluing scripts on the surface, making it incomprehensible for a sane person.

> Wanna build an fs for the device with a local storage and external ssd, good luck to figure out how

I never had to do anything even closely similar. My embedded systems have just one rootfs.

I would start looking here https://www.yoctoproject.org/docs/3.0/mega-manual/mega-manua...

> Which filesystems could I use? How to know that?

A simple

  git grep IMAGE_FSTYPES

brings you up a long list. File meta/classes/image_types.bbclass seems particular promising. And there it is

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/cl...

I'm very far from claiming that Yocto is easy to understand. But you get much more than you pay for. What alternatives do you suggest with a similar functionality?

Do you think the Linux kernel documentation is better? The systemd source code easy to grasp? Still, they are widely used. I tend to say the price you pay for those is that you need to employ an expert and the expert needs to spend quite some time on it.

If you buy a product you are entitled to request something from your vendor. Good luck if you are a small company. I even tried when working for a 50,000 people corporation in the past and we got very little after many years of paying huge fees.

In open source it's take it or leave it. Preferably take it and contribute to make it better.

I wish they'd generate toolchains I didn't have to install in order for them to work, like Buildroot already does. With Buildroot I can just tar/gz the toolchain and libs and integrate them into a build system. This lets me support cross-build in a relatively pain-free way. Buildroot toolchains do not care where they are located on the filesystem, so e.g. Bazel can download and unpack them without any issues and they work fine. In contrast, Yocto generates toolchains that have to be _installed_ and in which the installer has to _rewrite the locations of dynamic libs_ during installation. This makes Yocto toolchains effectively hostile to build automation.

If you find yourself wrestling with this in the future, you might try experimenting with Yocto's SDK generation capabilities. Most image recipes have a 'populate_sdk' task that will generate a self-extracting shell script that includes a full cross-compilation toolchain and a set of libraries for the target. You can also add custom components to the SDK very easily.

Our CI system generates the SDK along with every build of our Yocto image. Another CI job grabs the SDK and uses it for some out-of-tree compilation work. It's very straightforward to automate that way.

Yes, and that shell script can't be simply unpacked. It has to run, and when it does, it rewrites paths to libs that the toolchain needs to pull in so that they point to the correct installed locations. In contrast, Buildroot toolchains require that only relative paths match, and do not require that absolute paths are embedded into the binaries and *.so's.

You're right, of course - I wouldn't say that Yocto's SDK generator is perfect. But it does work fine, and you can install the SDK anywhere you want.

From a CI system's point of view, what's the logical difference between [tar xf toolchain.tar.gz] and [./install_sdk.sh -yd .]?

Thanks to the Yocto team for a great release. I'm excited to start playing with the new hash-equivalency support, which is the biggest addition in this release IMO.

Keep up the good work! Yocto is an amazing project, and I'm a heavy-user in my day job.

I was really hoping for big usability/learning curve improvements with 3.0. I've tried using it and other engineers I know have tried. It just ends up being a complete cluster.

Yocto steep learning curve is a fact.

https://www.youtube.com/playlist?list=PLD4M5FoHz-TxMfBFrDKfI... is an attempt to help.

Disclaimer: I have not watched them yet.

When I needed an embedded Linux, I wrote a 3-pages long shell script to customize vendor-supplied Debian image. Most of these 3 pages are sudo apt-get purge commands (I don't need much except kernel).

Pretty sure I would have spent at least 10x more time building my own image from scratch.

Got excited then, because I thought it was a follow up to the Yocto 808 [1]

[1] http://www.e-licktronic.com/en/content/25-yocto-tr808-clone-...

I sometimes use Yocto at work to rebuild certain recipes in an embedded Linux distribution, but I'm really confused about what "Yocto" and what "Poky" mean. If anyone knows and has time to dispel my confusion, that would be awesome. (I'm trying to answer a bunch of my own questions in here -- please let me know if I got anything wrong.)

"BitBake" is easy -- that's the command you run to rebuild "BitBake recipes" (a recipe is something like a Makefile, except that it's more "meta" -- it defines what to run in order to build software packages, i.e. how to find and build a source directory that may have Makefiles/CMakeLists/whathaveyou in it -- so I think it's a bit like e.g. an RPM spec file). So you run stuff like 'bitbake -c clean packagename' or 'bitbake -C compile packagename'. (BTW, I often want to do things in a single command -- is that possible? Reloading the whole cache every time gets old after a while.)

Fine okay, "Yocto" is presumably the name of the entire project. But what's Poky?

> Poky is a reference distribution of the Yocto Project®. It contains the OpenEmbedded Build System (BitBake and OpenEmbedded Core) as well as a set of metadata to get you started building your own distro.

Hrm, what's OpenEmbedded Core?

> OpenEmbedded-Core is a layer containing the core metadata for current versions of OpenEmbedded. It is distro-less (can build a functional image with DISTRO = "nodistro") and contains only emulated machine support.

Hrm, okay. (BTW, for the people who don't know -- "metadata" most likely refers to a collection of "BitBake" recipes (i.e. a directory containing loosely related BitBake recipes), so the "core metadata" would likely be the recipe for at least the Linux kernel.) (BTW2, in BitBake, you customize recipes using "layers". The first layer contains recipes that build things in a generic way, and then you add other "layers" that contain "bbapend" files that override variables or commands in the lower-layer recipes to build something the way you need it.)

Okay, so is the "OpenEmbedded Build System" still part of the Yocto project? From Wikipedia: "BitBake is co-maintained by the Yocto Project and the OpenEmbedded project." Okay, that explains that a bit. But why are they separate in the first place?

Why does "Poky" have such a weird name, and what kind of recipes does it bring to the table? Looking at the repository (http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta), that's quite a lot of useful packages.

One other question that just popped up in my mind: are there any Desktop Linux distros that use this as their build system?

> Why does "Poky" have such a weird name

"Poky" in English means "small". The idea being that the distro is engineered to build small OS images for small embedded devices.

Though that definition is correct (had to look it up) the more common one seems to be "annoyingly slow". at least that's the only one I actually knew or ever heard.

It might be a slightly older phrasing, but to match your anecdote: I've never heard of poky meaning slow.

> It's located in a pokey little village in the middle of damn near nowhere.

You've never heard of a slow-poke? Similar usage, you can be pokey or you can be a slow-poke, both mean slow. The "small" definition is new to me. Seems like difference between British/American English.

> You've never heard of a slow-poke?

Sure, but that doesn't mean "slow slow". The poke part of that that means something similar to "yoke".

A slowpoke is something that's slow and dragging you down with it.

It's not exactly a positive term though is it. It's a strange choice if this is truely the reason.

All the names of Yocto (or better openembedded, which is it's ancestor) have to do with food. There is bitbake, cooker, toaster, and what not.

Pocky is a brand of chocolate coated bisquit sticks. (Not very international, I have not seen them.) But using a trademarked (I assume) name might end up in trouble, so they changed the spelling but kept the pronunciation.

> are there any Desktop Linux distros that use this as their build system?

I don't think so and I think that is not the goal of the project. It is more like a build system to meet the requirements of "I need to build a set of packages for architecture X and additionally a host SDK and a deployable system image", so it lets you whip up a recipe with your requirements and off you go.

OpenEmbedded and Yocto are separate entities in time. The OpenEmbedded project started in 2003, Yocto in 2010/2011.

When you "use yocto", you're likely just using parts of the OpenEmbedded project. Yocto takes all the flexibility of the OpenEmbedded project and packages together a functional setup (called Poky) that can be more easily customized to build custom images. (There are non-poky, non-openembedded components of yocto, but I'm not familiar with them).

The default packages shipped with Poky are I suspect just the minimal set required to get a network connected device with a GUI interface up. Remember that it's default setup is effectively suppose to be a "hello world" project.

> Why does "Poky" have such a weird name

The pronunciation is even more weird: "Pock-ee"

https://www.yoctoproject.org/docs/3.0/mega-manual/mega-manua...

src: https://linuxfr.org/news/yocto-zeus

Reply on news.ycombinator.com