Hacker News Reader

Comment by palata | Hacker News Reader

Yocto is pretty great! Unfortunately I feel like it gets a lot of criticism, but usually from people who haven't gotten to learn it. Like "I had to spend 2h on Yocto and this thing suuuuucks, I threw a docker image there and called it a day".

Which is a pity, because when used correctly it's really powerful!

From the article, I can't help but mention that one third of the "key terminology" is about codenames. What do people have with codenames? I can count and easily know that 5 comes after 4. But I don't know how to compare Scarthgap and Dunfell (hell, I can't even remember them).

View on HN

Part of why it gets so much criticism is that Yocto’s learning curve is pure brutality.

Out of the box configurations for Yocto images and recipes are fabulous.

Trying to modify those configurations below the application layer… you’re gonna have a bad time. Opaque error messages, the whole layers vs recipes vs meta issues, etc. I also can’t shake the feeling that yocto was made to solve a chip company’s problems (I.e. supporting Linux distros for three hundred different SOCs) rather than my problems (I.e. ship working embedded software for one or two SOC platforms).

I’ve had a lot more success with buildroot as an embedded Linux build system and I recommend it very highly.

> pure brutality

And that's not hyperbole.

It's an odd mix of convention and bespoke madness. The convention part is that you set up a few variables and if the build system of the software is a good fit to common convention, things will just tend to work.

The bespoke madness comes in when there are slight departures from common convention and you must work out what variables to set and functions to define to fix it.

There are parts of the build system that are highly reminiscent of 1980s era BASIC programming. For example, I have seen build mechanisms where you must set variables first and then include or require a file. This is analogous to setting global variables in BASIC and then calling a subroutine with GOSUB because functions with arguments haven't been invented yet.

I've done both and I'll add that the one thing I miss about Yocto is that it could package up an SDK with installer that could be deployed on a different machine. With a single install you have the correct crosstools, libraries, and headers to build directly for target. And when we used to develop with Qt that was a huge advantage in helping others get started.

But now I use Buildroot and I get things done without all the extra anxiety.

Fair point but buildroot reached parity with that feature by allowing you to zip a toolchain and then point to it as an external tarball.

I have found that whatever my Yocto question is, reading (edit: bitbake -e and) 1-3 pages of pretty simple recipe and / or Bitbake code (Python) gives me the answer. Apparently, none of the people who complain have gotten far enough to run into the real problem... Bitbake is a shitty build system. Its dependency modeling is incomplete, so if you change some variable that influences many packages, you have to manually rebuild them. I just resorted to rebuilding everything after making that kind of change.

Ha kind of ironic that there's a comment above that says this is an advantage over Buildroot.

In my opinion the task Yocto and Buildroot do means it is impossible to have reliable incremental builds. They're simply wrangling too many other unreliable build systems (autoconf, make, etc.). For example OpenSBI's Make based system doesn't do incremental builds correctly. I always have to build from scratch.

The only way it could really work reliably is if everything it builds used Bazel or one of its brethren.

At a client they use an obscure system called e2factory to build images and that one does model dependencies correctly. It creates a little chroot environment for every package build and errs on the side of rebuilding if in doubt. It's probably less flexible than Bitbake with Yocto, but incremental builds totally work. You might need some all-encompassing integration to get more fine-grained dependencies, but tracking what goes into a build is possible with standard build systems (mainly autoconf / make and CMake / Ninja in this case).

That sounds like a good design.

Well, it has downsides, too - one needs to specify all dependencies for every package. That includes transitive dependencies, so some of these dependency lists look pretty weird. "I need video codecs to build the mouse driver?!" (not a true example, but along these lines)

But with that information, everything that goes into a package goes into the build signature. Environment variables and all.

For the typical use case (building a highly customised, target specific image for a pre-defined applications), what’s the main advantage of using Yocto over Gentoo (eg: creating images with Catalyst)?

It seems that they’re both well catered for that goal, but Gentoo really is designed for you to dig in deeper if you want (rather that trying to abstract everything away).

Yocto cross compiles. I don't think Gentoo can do that?

This is truth. I've done a lot of work bringing up custom boards with both Yocto and Buildroot and I rather like Yocto. But yeah, first project with Yocto was painful. It's rather well documented, but one really needs a good idea of what's going on to really know what you're even looking for.

That said, once you get it figured out, it's very flexible and largely logical. :)

IME, the documentation is kinda useless when the going gets tough. But the code is surprisingly easy to follow.

The docs are good in the "I think there should be a way to do X" and if you have a good sense of how Yocto is organized, you'll know where to check if X exists. That's...not a trivial thing.

True. For the most part, the documentation tells you what you can do in general terms and where (which file / variable). That is useful, but not enough to know how to do it.

"Part of why it gets so much criticism is that Yocto’s learning curve is pure brutality."

At one time when SoCs were RAM lean... and build specific patching, stripping and static linking was considered an acceptable tradeoff in the yocto build systems for IoT etc. The use-cases are extremely difficult to justify these days with 256MB of ram on a $5 SoC...

However, the approach was commercially unsustainable from maintainability, security, and memory-page cache-hit efficiency metrics. It should be banned given it still haunts the lower systems like a rancid fart in an elevator. =3

From experience, none of the difficulty of Yocto comes from the fact that it strips binaries; it builds stripped packages and puts debug info in separate -dbg packages, which is super standard in the Linux world.

Yocto doesn't do static linking unless you specifically ask for it, libraries end up as .so files in /usr/lib like on all other Linux systems.

When Yocto carries patches, it's typically because those patches are necessary to fix bad assumptions upstreams make which Yocto breaks, or to fix bugs, not to reduce RAM usage.

I don't understand where you're coming from at all.

"I don't understand where you're coming from at all."

In time you may, but perhaps you were confused about the primary use-case context bringing up small linux SBM. The mess Yocto can leave behind was not something manufacturers prioritized, and there are countless half-baked solutions simply abandoned within a single release cycle. Out of date package versions, and storage space-optimized stripped/kludged binaries are the consequences. Historically, the things people did to get the minimal OS on flash also meant builds that are not repeatable/serviceable, buggy/unreliable (hence custom patches), and ultimately in mountains of e-waste.

My point was Yocto has always created liabilities/costs no one including its proponents wanted to address over the long-term. Best of luck =3

I might have been unclear. What I don't understand is how binary stripping and custom patches is associated with SoCs with low RAM, or why you claimed that it does static linking when it doesn't by default.

It was mostly the low-end IoT and router markets that was the major driver behind space-optimized image builds. i.e. the various tricks people pull to get leaner builds made maintenance nearly impossible. On chip RAM was the initial constraint that fell, TLC flash became inexpensive, and ARM option performance hit practical levels. i.e. it became possible to install normal environments with a simple port.

The other point I was stating was today the low-end chip justifications no longer makes economic sense. The kernel maintainers already deprecated 32bit years ago.

"All software is terrible, but some of it is useful..." but if proves a liability, than it is just terrible. lol Have a great weekend =3

You're not responding to what I'm saying, the questions I'm asking or the mistakes I'm pointing out. I'm done.

Thus, we agree the bodged binaries were a mistake, and thus have reached an awkward understanding. Your use-cases might differ, but it does not change what the artifacts look like from a optimized low-end Yocto project.

Best of luck, some of my most prized friends took 3+ years to agree with my perspectives. You should know I hold you very high regard =3

I don't know what you mean by "bodged binaries".

Upon the code audit, we saw heavily modified package builds where the linker was setup to strip off parts of partial libraries (far beyond what "strip --strip-unneeded" or even "-O3" would dare.) This meant the normal versioned ecosystem was always going to be out of date, potentially unstable, and difficult to curate properly.

Keep in mind there is zero justification for this trick unless constrained by minimal flash storage. My point was the situation shouldn't have featured in years (except in IoT garbage products), and using a standard build makes more sense these days even if the SoC costs $0.80/pc more.

YMMV, good luck... =3

Buddy what the fuck are you talking about

Yocto launched in 2010

Buildroot launched in 2005

Both of these ecosystems coexisted in the era of sub $100 embedded Linux dev boards with way more than 256MB RAM

Yocto has no excuse for making toolchain and system configuration modifications as difficult as it does.

There is a big difference in just about everything relating to selling something with a sub $10 BOM and something approximating a “sub $100 dev board.”

The difference in unit volumes drives wide variances in tolerances of additional development difficulty/cost.

Indeed, the economics of chip component choices at scale change development priorities. Depends on the use-case, and how much people are willing to compromise on the design. Performant SoC and Flash memory are no longer premium budget choices.

Some people seem irrationally passionate about the code smell of their own brand. =3

It's powerful but bitbake wasn't so much designed as emerged from a primordial soup, it's easy to go completely insane trying to debug it due to the amount of action-at-distance recipes and layers can create. (try playing the "where did this compile flag come from?" game)

I don't disagree, but wanted to note- if you're ever stuck trying to trace a value, you can see everything that went into its calculation by using "bitbake -e".

bitbake -e <recipe> is super useful for that game. It dumps out a complete history of where all variables were set/changed, and their values along the way. I also use it to do what I call "variable shopping," where I roughly know what path/name content I need, but not what the variable it is in is called.

Yeah, I was a bit scared off by it and both the terminology and the curious mixing of Python and bash can be a bit confusing. But it’s powerful and also very extensible without (generally) having to fork upstream layers.

I'm honestly impressed by how...well it works. Considering it's building an entire, totally custom Linux distro from scratch it requires a surprisingly little amount of hand-holding.

I agree. I don't understand how people prefer buildroot. Buildroot feels like an adhoc system of glued together Makefiles, whereas yocto actually feels like it was built for purpose.

Yocto feels like a ball of mud duct taped together, but thankfully has good documentation. It reminds me of CMake. Buildroot is nice for relatively simple situations. Nixos is arguably better than both.

Their idiosyncrasies may look similar, but CMake has a much stronger skeleton of core algorithms and data structures for a build system than Bitbake. Specifically, as I mentioned in another reply, Bitbake does not model dependencies correctly. CMake does.

Can you elaborate a bit on the dependency-handling topic? I've always thought that Bitbake's dependency handling worked pretty well. It only has package-level granularity, but is quite good within that context.

As of five years ago when I last used it:

Bitbake doesn't model all changes that affect packages, so after certain changes, some packages that should be rebuilt, aren't. It is especially prone to happen when changing Bitbake variables (example: MACHINE_FEATURES), and these are a quite common way to change things about the image being built.

Reply on news.ycombinator.com