So you want to build an embedded Linux system? (2020)
jaycarlson.net
jaycarlson.net
This article is awesome and has a ton of great knowledge, but it’s a little dismissive of openembedded, and if you click through to the previous thread you’ll see a lot of people come to it’s defense.
I have been doing embedded development for 30 years, and Yocto is the best build system that I have used for managing large embedded systems. If you need to professionally manage multiple distributions across menagerie of different host and target machines, there simply is no other meaningful choice that operates at its scale.
I first encountered BitBake before either the OpenEmbedded or Yocto projects existed. Since then, I have done professional embedded systems consulting for dozens of platforms, using Yocto almost exclusively in recent years. I have done everything with it, from simple board bring ups to implementing re-usable BSP layers used by multiple product families. I might be a little biased here, but I have used all of the other popular embedded build systems too.
Other choices still have their place for smaller and simpler projects, just as one might choose something smaller than Linux for the kernel. However, Yocto and Linux have attracted unparalleled open source development and support resources. Other projects simply cannot hope to match their established output. These projects will not go away any time soon, which means LTS releases and stable infrastructure to host them, regardless of individual players or projects coming or going. What other choice exists with that kind of backing?
For commercial projects that grow beyond a certain scope (i.e. most modern full-featured embedded systems), other choices simply do not match its existing capabilities. Of course, you can always fork a full server/desktop Linux distribution like Debian, but dressing up a standard distribution in embedded clothing often requires more customization. That comes with develop and maintenance costs, though I admit the lines do start to blur at a certain point.
The BitBake/OpenEmbedded/Yocto/Poky ecosystem is huge and poses a long and steep learning curve, but it offers customization and extensibility unparalleled by any other offering that I have had the (mis)fortune of using in my career. The same is true for the Linux kernel ecosystem, right? Them's the trade-offs we make.
If I got a codebase that uses Yocto/Openembedded, the first thing I do is to find the related patches there and convert them to buildroot/debian/openwrt/LTIB(old stuff)/etc depends on the hardware spec. I wasted too much time when there is a bug in Openembedded that could take forever to debug, layer over layer, override over override on those meta mazes, it always feels like you're fighting with openembedded/yocto itself instead of getting your work done. I since have been avoiding openembedded/yocto at all costs.
It's over engineered, over rated, great for consultants to charge hours, terrible for companies(except for big guns) to deliver product fast.
Sorry I have a strong opinion here but it just wasted too much of my time in the past.
Do you have one board that you need to bring up in isolation? Buildroot is great. It's easy and fast. Do you want to share configuration across multiple designs? Buildroot hits its limits pretty quickly there. And Yocto really starts to shine.
I worked at a company that used Buildroot to support 20 different boards, all in the same repo. The boards shared some level of common configuration across the whole repo, and groups of boards also shared some common configuration.
The entire thing was managed by the C preprocessor as a pre-Buildroot step, and it was a gordian knot of emergent complexity.
The catch is that Buildroot doesn't give you any tools to enable shared configuration, none at all. Buildroot doesn't know how to share build-products across multiple images, and doesn't even know how to _build_ multiple images or keep them separate without having to clean your workdir. It doesn't even track file dependencies, let alone configuration dependencies. It's terrible at knowing when a package is stale.
Yocto is much, much better than Buildroot for handling these kinds of use cases. Images are just recipes, like any other kind of recipe. You can build 8 different images by just asking for each of them by name. Yocto won't rebuild anything that can be shared, and your images can be subclasses of each other which just add or remove a few packages or a bit of configuration.
Yocto is also better at tracking/managing customizations. In Yocto, it's easy to express something like "use the upstream OpenSSH recipe, but do something custom for the config step, and add an extra post-build step, and add these 3 patches, but only add patch 3 on some designs, etc".
In Buildroot, you have to do this in a global namespace using macro-processing in GNU Make. You have no control over the parse order of included Makefiles relative to the upstream files. It's messy and difficult to modify anything, and you have to understand each recipe's quirks and hooks. And the only programming language you have is GNU Make.
In Yocto, there is an inheritance system that lets you use .bbappend files to override the behavior of any recipe you want. You can also stack multiple append files across different layers. If you need something exotic, you can express it as parse-time Python. Yocto is more complex up-front, but it pays off IMO.
network gears, go openwrt, general spi-flash based boards, buildroot it, anything with a SD card and 256MB+ RAM, just use debian, if you're a BSP vendor like windriver, take on yocto then(which is why Intel 'owned' Yocto when they also bought windriver then)
What's your take on the various *BSD distros in the embedded space? A lot of BSD proponents like to talk about the stability and minimalism (a proxy for resource-efficiency I guess) of BSD in comparison to Linux.
Is there any reason to reach for BSD in an embedded application these days?
Generally, esoteric hardware support also will be better in Linux, so BSD would likely require porting or writing more drivers when bringing up a custom board.
That said, BSDs have not my strongest area, so I would be curious to see others’ experiences and opinions.
I don't really see it in the article.
I use OpenEmbedded for my projects, but I never recommend it to beginners as the place to start. Buildroot is unquestionably easier to get started with and use for one-off projects (unless you're trying to use vendor-provided Yocto/OE layers, in which case obviously it's not an option).
The article's conclusion about Yocto is spot-on, in my opinion: If you're quickly iterating, Buildroot is the way to go.
My all time favorite SoM was the CHIP Pro, but sadly at $7 (or was it $13?) it was indeed too good to be true and the company went out of business.
They're great. In my head they were "something that had the stuff that annoyed me about RaspberryPis fixed".
I should dig some out and see if there's a way to update the OS to something recent-ish.
I think I've still got a cluster of 4 of them, 3 running RTL SDR and one analysing the signals and pulling interesting data out. (Last data file I have here from that is Dec 2018...)
> My all time favorite SoM was the CHIP Pro, but sadly at $7 (or was it $13?) it was indeed too good to be true and the company went out of business.
The CHIP was just a cheap board based on the Allwinner R8, which is basically an Allwinner A13, which is in the same product family as the Allwinner A33 in the article.
There are actually a lot of cheap SBCs based on the cheap Allwinner parts out there. The CHIP had great execution and some novel in-browser upgrade features, but I don't know how they planned to make money by selling things basically at cost. Apparently they didn't have any idea either, because they spontaneously went out of business and disappeared.
Microprocessors are just the CPU, usually with the system bus wired out. Microcontrollers are combined systems, which include a processor and usually some kind of firmware storage and RAM. You can usually differentiate them by whether their pins are address/data lines or gpio-like.
> microprocessors have a memory management unit
The 8086 didn't. Having an MMU seems unrelated to the MCU/MPU distinction...
At least, this was my knowledge until now. Is this like "crypto" where the meaning changed while i was not paying attention?
Microcontrollers may or may not have storage for firmware. Requiring an off-chip flash is nothing new. Microcontrollers have SRAM, but so do processors. That's where the firmware runs before it gets the DRAM running. Microcontrollers may include a bunch of peripherals, but so do processors. Especially so if you refer to SoCs as processors, like the article here does. And SoCs stir it up further: they have tons of peripherals, and we also have SoCs with on-package DRAM. Discrete SRAM chips exist too, I figured there must be microcontrollers that can use them (but I haven't actually looked).
Yes, the terminology has evolved.
> Microcontrollers have SRAM, but so do processors. That's where the firmware runs before it gets the DRAM running.
Which microprocessor architecture does this?
"Emulation" evolved to mean virtual machine, now people are denying that wine or xterm are emulators.
"Operating System" evolved to require virtual memory and paging, now people are denying that MS-DOS was an operating system.
"Microprocessor" evolved to mean computing chips with MMU, im waiting for people to deny that the 8086 was an microprocessor because it does not have an MMU.
The infinite shitcycle of people improvising language instead of researching.
https://web.archive.org/web/20150928042254/http://wiki.wineh...
I mentioned the shift of meaning to "virtual machine" in the post you replied to, please read it more carefully.
Emulation and Simulation are different in some regard: emulation is only imitating some aspect of a thing, while simulation imitates a thing by using (usually physical) models.
xterm is a VTxxx emulator, and VT simulator does look like this: https://www.pcjs.org/machines/dec/vt100/
I totally agree with your main point though that emulation is a broader topic than is generally thought.
Also, back in the day emulators emulated the CPU for sure, while an API implementation wasn't never called "emulation". Ever.
2) SRAM is almost always on the die, not external. Here's a die shot of a random STM32 (cortex M): [1]. The SRAM is quite obvious.
x86 being the classic example. Here [1] is an example of a microprocessor board, you can see the MX support chips on both sides of the CPU. As you can see, the RAM (and the most of the address space-mapped things) is not on-chip. The bus is wired across the PCB.
[1] https://upload.wikimedia.org/wikipedia/commons/d/de/386DX40_...
All the RAM you see on the motherboard is DRAM. It's a totally separate thing and happens to be what the FSP initializes after it loads stuff into SRAM.
I regularly use the 1788 + a meg or two of DRAM to hold an LCD framebuffer. The LPC has a really nice interconnect between EMC and the internal LCD controller block and will drive the display all on its own (with a hardware cursor) once it is initialized.
> LPC is a family of 32-bit microcontroller integrated circuits by NXP Semiconductors.
Not a microprocessor. You are the second guy to pull this on me. Do people even read what i write? Did i write unclearly?
It has a x86 Microprocessor architecture, but it is not IBM PC compatible because it does not has an BIOS, instead using a more embedded-style RedBoot setup.
Maybe you should do a quick web search before talking shit.
"chipset" is very different than a traditional microprocessor chipset though on that. E.g. you don't have an exposed system bus outside the CPU, but rather specialized interfaces. (And ARM systems have evolved the same way over their history)
Traditional microprocessors are getting really rare, even though pretty much all the modern architectures started as them, and thus some use the term more widely. Some vendors use the differentiation between microcontrollers and application processors now, which also isn't a 100% clear line and explicit crossover models existing, but more useful today and avoids the fights over "what's a microprocessor today". (where "application processor" ~ "can reasonably run a full OS like Linux"). But that's also often limited to discussion in embedded use cases again, e.g. I don't know if they'd call a standard desktop CPU that or insist on some arbitrary level of "embeddedness"... Not that standard desktop CPUs don't end up embedded, but that's another discussion entirely.
For the article, i don't want to leave it standing as it is. Redefining "Microprocessors" as "having an MMU" is making communication about these topics very unpleasant, especially when its about Retrocomputing. Same pains as i ranted about in https://news.ycombinator.com/item?id=30278936 .
I've done this with PowerPC cores that no one would call microcontrollers.
https://www.coreboot.org/images/2/23/Apollolake_SoC.pdf
https://blogs.coreboot.org/blog/2019/07/17/gsoc-how-to-run-c...
https://9esec.io/blog/open-source-cache-as-ram-with-intel-bo...
The TLDR is that you need to get DRAM running before you can use it (e.g. by poking at DRAM controller's registers until it knows how to talk to your stick of RAM reliably and at a comfortable speed). Until then, you're stuck with SRAM. On ARM SoCs and Intel Apollo Lake, you use addressable SRAM. On most other x86 systems you use cache as RAM. Cache is just SRAM though, and the only difference between ARM and typical x86's case is that the latter doesn't let you directly address it.
You can find examples of DDR DRAM configuration in u-boot sources, and presumably in coreboot too.
As far as I know, all of them. It's possible that there are historical systems where DRAM configuration is fixed, but I wouldn't know about it. Any modern system requires rather complicated configuration to get the DRAM going.
Excluding the caches, this is how x86 functioned for quite a while, i don't know until when.
Microprocessors don't require internal RAM to bootstrap, because they expose their bus and its simple to map a ROM to the starting address, which can be executed from directly.
Which bus is "the bus"? SoCs and microcontrollers alike expose a bunch of different buses.
I'm assuming Naumann arch, with instructions and data on a single bus.
If anything, the things people today refer to as microcontrollers have more to do with your legacy microprocessor than they have with a high end CPU or SoC.
What's the right term to use?
The BIOS starts running on the x86's internal SRAM.
https://stackoverflow.com/questions/63159663/how-does-bios-i...
All of them. They have to. Sometimes the trick is using cache as RAM. Some Intel CPUs do early memory init this way, for example.
You need some memory somewhere to run the complex DRAM init procedure.
We know definitions change over time because slang words come and go and someone's vocabulary can be used to identify people online or identify their knowledge.
The problem with these definitions is a control device and computer are two different things, so I wonder if he is applying an engineer's perspective/definition to the 8086 even though many know the 8086 were used in the early computers. This is a bit like arguing whether a microprocessor is a microprocessor because of the inclusion or exclusion of one or more instruction sets, ie a basic intel cpu with a minimum instruction set or a xeon intel cpu with a much more extensive number of instruction sets.
Or are you picking up on an AI masquerading as a blogger and have highlighted something an AI wouldnt pick up on from online technical resources?
Obviously this wasn't always the case, e.g. a good 15+ years ago I put ucLinux on an old ISP router I had lying around for fun, and it was an ARM946 without an MMU. But that class of device no longer exists today for all practical purposes, and so it's easy to use the existence of an MMU to decide what to call them.
The 8086 may have been a microprocessor, and so was the 80186, but you know where you find 80186 cores today? Microcontrollers. Specifically, some video processing chip manufacturers (e.g. stuff that goes into monitors and DP-HDMI adapters) seem to like using V186 cores still.
So really the more practical terminology difference should be SoC vs. microcontroller.
Our of curiosity - does anyone still sell a (by strict definition) a true "micro-processor"?
Digi-Key still shows the 750CL in stock, but it's obviously an obsolete product at this point.
Microprocessors going back to the Intel 486 have more on-chip RAM than many microcontrollers sold today have. CPUs have built-in firmware ROM for loading boot code. Also, CPUs haven't had a system bus with address/data pins for like 30 years — somewhat ironically, the only chips that have that are microcontrollers. Northbridges haven't existed for 15 years, and southbridges are basically just PCIe-to-USB/SATA/Ethernet/I2C adapters. And many mobile CPUs that Intel and AMD make have on-chip I2C, SPI, UART, GPIO peripherals.
In my article, I'm trying to differentiate between processors that can run a modern multiuser operating system like Linux and microcontrollers that runs bare-metal code, since that's a question I get asked a ton. After carefully considering all the options, the MMU seemed like the most obvious place to draw the line. Every CPU in the last 40 years (since the Intel 286) has had an MMU, while no microcontroller ever made has had one. I'm not sure why that seems so unreasonable to you.
Surely odd, but tolerable. But what is worse that people will read your article, think the MMU is what makes a microprocessor. And upon hearing that the 8086 has no MMU, they will deny that it is a microprocessor. Which would be a logical conclusion from your article.
I listed some other words here that suffer from similar issues https://news.ycombinator.com/item?id=30278936
JimTCL is very fun for that, and it would run on limited machines.
Ok, not totally TCL compatible, and no TK, but you have (WIP) SDL2 bindings, and SDL2 would work perfectly on a display handled by fbdev/KMS.
> but blending and other advanced 2D graphics technology can really bog things down
SDL2 can be really fast at low res displays such as 320x240. If not, EFL from Enlightenment for sure could that fast enough.