Moving to a RTOS on the RP2040
blog.brixit.nl
blog.brixit.nl
Given a lot of Arduino these days have mbed or freertos under the covers, with some way to expose it, that may have been a better route for the author's style.
Zephyr is easy to use (Clion also has good support for it), but like, you can't just choose not to install the toolchain and expect it to all work.
It also definitely supports the Pi Pico - i've used it on it before, no issues.
A simpler rundown of these RTOSen would be:
1. FreeRTOS - supported by roughly everything, but drivers and such are mostly per-SOC/device, which is a pain in the ass. The APIs are not very user friendly, but you get used to it.
To give a sense of level - if you wanted to use bluetooth with FreeRTOS, you get to find your own stack.
2. Zephyr - supports real hardware abstractions and supports most SOC. You may have to do a little board work.
If you wanted to use bluetooth with Zephyr, it has a bluetooth stack you can use, you may have to add a little support for your HCI.
3. NuttX - not great support, but very cool if you can get it working - not really highly supported by industry yet.
I never got far enough into NuttX to try bluetooth in NuttX.
There is also mbed, but we'll skip it.
In practice, people in the RTOS world will usually go with what the SOC vendor supports. So if you are using Nordic stuff, you are probably on Zephyr. If you are using NXP stuff, probably FreeRTOS, etc.
That is how they get good support.
That has not been my experience.
From where I sit, Zephyr has very pretty marketing material.
Behind all that snazz, Zephyr is outrageously bloated, extremely slow to compile and wildly difficult to get up and running.
If there is board work, at that point i'm playing with making the board work.
I started with platformio's zephyr support and then moved to clion as platformio kind of died.
I have never, and would never, try to use west as the main build system of my project directly.
As for bloat/compilation speed, i guess it all depends on what you are trying to use it for. As per the other thread - if you are trying to use it as "an RTOS that is a less hacky version of arduino stuff", i think it works great.
If you are trying to use it for "i have meaningful hardware constraints and need to keep track of every byte", i doubt it would work well compared to FreeRTOS.
I think the limit is probably "i am using nordic boards to develop bluetooth stuff".
I do agree they try to sell it for a lot of things.
They seem to only care about Espressif these days.
Zephyr support has not been updated in ages (2021)
Patches submitted to platformio arduino to add Giga R1 support have gone unreviewed for years (I get poked every so often because i did some work on it, so i see the github comments).
The only things that seem to be in okay shape these days are ESP and STM32.
Given the other options at this point, it is nowhere near as useful as it used to be.
ESP had a pretty public dispute with them (payment wasn't explicitly mentioned but can be inferred).
As for why, it is probably because MCU makers are a bit nearsighted when it comes to the software side so you get little or no help from there. The strategic decision makers don't understand software. (That's not my assessment, by the way, but that of people I know who either work for one of the major MCU makers or has worked there).
So they do their own thing, they don't do it particularly well, they don't see that they don't do it particularly well or why it is even their problem. I also think that they are a bit afraid of common tooling - possibly because they think it is giving up control or robbing them of the opportunity to differentiate themselves (which is tragically funny since most can, at best, hope to be no worse than the competition).
You'd think that if lots of players can agree on using Zephyr it shouldn't be hard to make them agree on supporting sensible common tooling (akin to PlatformIO), but then again, these companies don't really get software.
(I was tempted to say "modern tooling", but then I remembered that I hate it when people use the word "modern" so Dobby had to iron his hands and delete it).
I have never encountered a single project where we had to do firmware for an OEM device where developers didn't struggle with Zephyr. Not once. Nor have I actually met any of those mythical developers that do actual firmware work on shipping products who think Zephyr hardware abstractions actually help. I'm not denying they exist. I just haven't encountered any in the past 5 or so years.
As an embedded developer for over 25 years, I would never use the provided hardware abstractions from one particular vendor. Typically, you use your own abstraction layer over multiple vendors BSPs/OS/libraries.
One of the driving principles behind embedded software is minimalism because you pay real money for every byte and cycle. That leads you to creating minimal abstractions for your particular use case that still gives you the flexibility that is also often required (multiple sources for processors for example).
In turn, that means a focus on premade software to reduce Dev time rather than optimizing every byte.
I often use C++ with zephyr, which also pretty much immediately disqualifies me from that club :)
Though i've done it plenty of times with C only and nordic boards.
Amusingly, I got into it years ago because I was porting my dust collector controller to an RTOS, and it was very easy to port to zephyr (I had to hack up the C++ threading support, but the hardware was very easy to make work).
But i've since used it for everything from the "the keypad and receiver that is on the gate to my house" to embedded devices that transmit flow data, etc. I have had to fix bugs in the hardware abstractions for LoRa (interrupt based packet reception did not work on all chips), and do things like add hardware acceleration for LVGL, but overall the abstractions work pretty well for me. Having had to build themselves in FreeRTOS, it's just not worth my time.
I view it as more of a "less hacky arduino" than "serious thing i would base my life on".
At most, i would use it for nordic boards to develop bluetooth stuff (because they have such good support for this).
Anything that requires real serious guarantees i'll use PLC hardware, not zephyr. But i'm in the machine world, so this is like "make sure spindle has stopped moving before allowing tool release" type stuff that needs to be deterministic, but is also fairly simple. It does not require more than PLCOpen or ladder.
I mainly used ESP32 through platformio (which used the arduino compatibility layer of ESP-IDF), and then through the rust support.
I did use ESP-IDF directly for one project, but only for a small time. For that application, it was just way too high power and too finicky (it was easy to crash it by doing normal things - admittedly, it could be that there are just a lot of crappy ESP32 boards out there)
I did find ESP-IDF a lot nicer than either platformio, or the rust support, for sure. It was nice having regular old cmake and stuff that just worked for you, without worrying about it. For a simple app, i would use it over zephyr, no issue.
But zephyr's abstractions have bought me stuff. For example, I have learned how to make zephyr's MCU management subsystem do OTA over bluetooth across any device that supports mcuboot (IE everything). So like, i don't worry about needing to use each manufacturer's OTA support. It also works with nuttx if i ever go that way. It's hard to give stuff like that up for my use cases. I can walk near my gate transmitter/receiver, or near a machine, or near anything else i've built, and using the same tools, make sure the device is working and update the device with an image, without needing to worry about or keep 75 types of OTA tools and keys around :)
(I could make it work over wifi or lora or ... easily too, but i don't really need to).
These devices do not all use the same SOC - the gate transmitter is a very low power sleep device (small battery will last about 21 years at current rate). The gate receiver is not. The embedded sensors on machines are yet a different SOC (nordic), with other types needing a little more power being higher power STM's.
There is even an NXP device around somewhere from when i was experimenting with LVGL.
It is possible to make this kind of thing work with freertos (they have a 3 year old not-updated labs project to support mcumgr, for example). But i don't want to have to combine my own pieces, etc. This, of course, is basically the point of FreeRTOS, so it's not a surprise i don't go that way :)
ESP-IDF obviously provides it's own OTA layer, though it leaves transmitting data and management to you. I did build an OTA over bluetooth impl that worked pretty well (updates at about 50-200k/s) before i moved entirely to the zephyr one.
PLCs are awesome for what they are intended for, and they work very well with each other/equipment made with PLCs in mind. but they are an immense pain to "integrate" into an existing non-industrial system. They are basically in their own world, with tooling that's specific to the PLC world, with very little open source support.
I agree that for some one off things like controlling a CNC, they might be more useful than a Frankenstein esp attached to some servo controllee though.
You can easily get locked in. For anyone else who is just sort of lurking (you already know what i'll say).
Codesys is basically the standard here, and the way to "avoid" lock in - it has the most hardware support, and lots of vendors use it under the covers. This gives you some standard. It supports IEC 61131-3, happy to export it as text or whatever, and i've actually moved code between implementations.
You can use Codesys Control RTE on anything that runs windows or linux to get soft-realtime support.
Twincat is similar (it was based on codesys at one point but no longer).
Integration is, as you say, a pain outside of industrial land.
But i will admit i am amazed that i have an entire CNC machine built out of ethercat servo drives, I/O, VFD, vacuum pump, pneumatic valve actuators, limit switches, etc. They all are from different manufacturers. But damned if it doesn't work perfectly, and i only have an ethernet cable running between 99% of things where it used to require a metric ton of cables and you were just flinging bits and analog current around between things. 6 If i bothered to update the spindle to ethercat, the only real physical I/O would be brake power relays (unavoidable) and emergency stop. I do have one modbus thing (dust control flow monitor) i use as well.
It also is pretty good at hiding complexity - I can link a variable to an I/O input or analog value or VFD status word, know that it will deterministically update. I can set up a structure of bits for each pneumatic valve and map it to the manufacturer's single I/O word and again, get deterministic two-way updating and not worry about it.
Now, can i get status out of this thing? Well, no, to your point, either something else needs to speak modbus, ethercat, profinet, ethernet/ip, pure digital i/o to it.
These days i could publish status to MQTT, but something like "expose an HTTP port that outputs a bunch of JSON" is totally uncommon, and will net you strange looks. It's like you are asking about cold fusion.
Don't even get started on controlling it from the other side through something like that.
But yeah, otherwise paying 500 bucks for a license to ladder program a PLC is not a pleasant thing.
In a way that would be impossible without the insular ecosystem that the PLC world has, but that really doesn't matter considering how well they do the job.
And yeah, I think the reason why it hides complexity pretty well is that they are meant to be field repairable by regular technicians who don't necessarily know a lot about the underlying systems. That also makes it super easy to use... once you set up everything that is haha.
Thank you for the pointers, I heard about codesys but I always assumed that vendors all used their own proprietary islands of standards and only paid lip service to Interop. I'm only passably familiar with Siemens PLCs so not super knowledgeable either!
Centroid (known for the Acorn board) released Hickory, which does ethercat servo drives but not full ethercat support. It runs the same CNC software as Acorn/et al.
Vital Systems makes an ethercat motion controller that interfaces with Mach4. They let you use any ethercat device you have an ESI file for, and map the inputs/encoders/output types to Mach4 data of various sorts (digital inputs, analog inputs, encoders, etc).
MachMotion also has an interface to Mach4, theirs is a soft realtime motion controller based on RSI's (very well known/used for robotics) motion planning.
Those are the more standard ones.
Twincat also has an NC interface that supports gcode but you'd have to make your own UI.
LinuxCNC can do ethercat, but i've never considered it :)
There are also more standard hardware solutions. Syntec's hardware controller can do ethercat, etc.
It also doesn't help that people keep using Python for tooling. Why would you want to insist on using a language that brings its own versioning problems and which will behave differently on each developer's computer? I've done embedded development for about a decade now (both as a hobby and professionally) and it is puzzling that people think it is okay to spend a week trying to make everyone's setup do the same thing on a project and then not see how this is a problem.
It is a problem. It is annoying. It does waste time. It is unnecessary.
Tools should be statically linked binaries. I don't care what language people use to write tools, be it Rust, Go, C, C++. I just wish people would stop prioritizing ad-hoc development and start to make robust tools that can be trusted to work the same way regardless of what's installed on your computer. Python doesn't do that. And it doesn't help that people get angry and defensive instead of taking this a bit more seriously.
That being said, things like PlatformIO are a step in the right direction. I know it is a Python project (and occasionally that turns out to be a problem, but less often that for other tools), but they have the right idea: toolchains have to be managed, SDKs have to be managed, libraries have to be managed, project configuration has to be simple, builds have to be reproducible and they have to be reproducible anywhere and anytime.
I wish more of the embedded industry would realize that it would be better to have some common efforts to structure things and not always invent their own stuff. I know a lot of people who work for some of the major MCU manufacturers and I am always disheartened when I talk to them because they tend to be very near-sighted: they are mostly busy trying to solve their own immediate problems and tend to not have a great deal of focus on developers' needs.
I'd much rather have proper tooling to begin with rather than have to pull out everyone's favorite roll of duct tape (Docker) to hide the mess.
But being possible doesn’t mean that anyone does it. Most containers I’ve ever seen are big messes. Possibly GPL-violating messes if the payload isn’t appropriately licensed. Certainly maintenance and provenance disasters. Unquestionably not reproducibly buildable and possibly not buildable at all.
> Possibly GPL-violating messes if the payload isn’t appropriately licensed. Certainly maintenance and provenance disasters. Unquestionably not reproducibly buildable and possibly not buildable at all.
If static binaries are on the table then none of these things can possibly be disqualifiers.
There’s zillions of options for doing this nowadays, many of them self-hostable, which is pretty cool.
I have a keyboard project that runs on an RP2040, and the firmware is in Rust. Here are the steps to flash it if you're starting with just the repo and no Rust toolchain:
(install rustup)
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ rustup target add thumbv6m-none-eabi
$ cargo install elf2uf2-rs
* Put the keyboard in bootloader mode *
$ cargo run --release
Platformio: curl https://.../get-platformio.py | python3 && pio run
Pipx: apt install pipx && pipx [build-script.py]
Pipenv: pip install --user pipenv pyenv && pipenv run [build-script.py]
Tools written in C / C++ / Go / Rust just don't have nearly as much fuss, and they run faster and are smaller too.
You can, of course, build static binaries (and people really should do that more often) and distribute tarballs, but then you might run into all manner of licensing issues. (Not to mention the tedious discussion about upgrading shared libraries under the feet of applications and praying it doesn't introduce hard to detect breakages).
One direction I'd love for distro designers to explore is if Linux distributions were to abandon the traditional splatterfest style of software installs, where an application scatters lots of files across several filesystems and depends on system wide libraries, and instead puts each application in a self-contained directory. Or rather, two: one for the immutable files (binaries, libraries etc) and one for variable state. That way you could choose if you want to do static linking or dynamic. Because any shared libraries could be part of the immutable application directory.
Oh yes, it'll result in a lot more disk use, but disk drives are cheap, and libraries rarely make up that much of your disk use anyway. But there is a fair chance you could address some of that with filesystem level deduplication.
You could also have systems for distributing shared libraries from trusted sources so applications do not necessarily have to include the shared libraries, but that they can be fetched as part of the install system. And then manage shared libraries locally with hardlinks or some form of custom filesystem trickery. This would have a few added benefits (opportunities for supply chain security, reducing size of distributed packages etc).
macOS does this only partially, but applications still depend on shared libraries and you have the horror that is the Library directories which are somewhat randomly structured, resulting in lots of leftovers when you try to remove applications. macOS has some interesting ideas, but the execution of them is....inconsistent at best.
The reason package managers don't work well is that they are trying to solve a lot of problems we shouldn't be having in the first place. And because nobody wants to simplify the problem they have to solve, distribution maintainers will keep writing package management systems that are a pain in the ass.
cargo flash --release --chip <name>https://github.com/bschwind/key-ripper/blob/main/firmware/.c...
Embassy has it's own HALs which makes it better at async and has also nicer ergonomic IMO
I’m using RTIC for the firmware on my STM32H7 based product (https://umi.engineering) and it has been a joy.
Worth noting, you don't HAVE to use the embassy HALs with the embassy executor. However, AFAIK, the only mainstream non-embassy HALs that supports async is the ESP32 HALs.
There's no technical lock in, it's just that everyone I've seen implementing async support (outside ESP32) tends to do it within the Embassy project today. That may change over time.
Rust and Cargo take the pain out of building and flashing to the RP2040 (or STM32, for that matter) - it's the most pleasant embedded setup I've ever used.
A lot of applications simply don't use the MPU. And then consider what you get from Rust memory safety and the reduction in overall complexity of the firmware without the RTOS.
Additionally, you aren't intended (for many situations) to use a single "main" Zephyr install, but to include what external modules you need in your project's west.yml. If you have a number of projects sharing the same Zephyr install that's a separate discussion but installing every possible toolchain/HAL is not the only way to do things.
And it has a POSIX API layer if you really need that.
Its certainly targeting a more enterprise audience (much like ThreadX was originally)
You have to pay to use it, price unknown other than maybe >= 5k$ per year?, and it looks like it will be prohibitively expensive (for my usecases at least) to have source access.
To me, those make it nonstarters.
Personally I find microPython to be an easier path. async/await based cooperative multi-tasking works well for me. Latest project driving 6 stepper motors and a variety of LEDS and scanning buttons - all in what appears to be real-time to the user.
It is still surprises me how many don't grasp how little we had available to us, and still managed to use high level languages.
I can't say that I'm too familiar with PIO but a tiny bit of state (literally a few bits) and a few bytes of code should be enough to make a good driver and offload the whole task from the CPUs.
I know RP2040 has dual CPU but motor control is often the job of a timer on other uCs.
On a microcontroller, you could use PIO or I think the RMT peripheral or even just interrupt-driven pin poking.
That's good for many use cases, although I strongly suspect it would fall over once you tried a very high step rate with microstepping (I run at 800-6400 step/sec) but I will try this out both for my lightweight test steppers (which use AccelStepper) as well as my microscope (which uses FluidNC). The scope will make it quite clear if we lose steps as I have a visual reference (the microscope slide has a calibrated fiducial marker).
I'm old-school. Start by reading datasheets and then try out the simplest thing that could work and iterate from there. In general, I find third-party libraries more complex than my needs require.
Took me a fair bit of reading microPython documentation and experimentation to fully grok the capabilities of the tasking model and event_loops. But once I got over that, it lead to clean, single responsibility class implementations.
As an architectural approach it aligns closely with what I am going for in embedded land, but in C with more pain. And is not dissimilar to what you do in Erlang/Elixir in hosted land.
Embassy looks to be a good choice in more memory constrained situations where you can't afford multiple stacks.
This has been the #1 cause of quality issues for my (commercial) projects.
If you start a project with a new chipset, with a new vendor - build a new VM, install the vendors tools (and only the vendors tools) in that VM, and do your builds from there.
Do your hacking on your own (non-VM) machine, sure. That's perfectly fine.
But ALWAYS use the VM to do your releases. And, for the love of FNORD, KEEP YOUR VM IN SYNC WITH YOUR DEV WORKSTATION.
Disclaimer: currently going through the immense pains of dealing with a firmware build that absolutely has to be fixed while the original dev is on vacation - nobody can get into their workstation, the VM prepared for the task is 6 months out of date, and the customer is wondering why they should pay for the whole team to do something that one very special programmer is the only one on the planet can do .. grr ..
That does sound like a PIA.
In many cases when you have smaller systems, you actually don't want your OS to do dynamic memory allocation, since it opens up for all kinds of bugs related to eg double-free, or use-after-free, or running out of heap, or too defragmented heap.
It's also easier to calculate memory usage (and thus be fairly certain it will be enough) if you allocate as much of the memory as possible as static memory. Hence why some coding standards such as MISRA-C disallows use of dynamic memory allocation.
Exactly what caused the lock-up here isn't delved deeper into, but it might be that there was not enough heap, or not a working malloc-implementation linked into the compiled binary (could be a stub), or could be that a non-re-entrant printf was linked in while they tried printf from several threads.
Printf isn't re-entrant, and they are calling it from multiple threads.
There are solutions with trade offs: https://interrupt.memfault.com/blog/printf-on-embedded
Everything in embedded is a tradeoff of space, performance, or cost.
If you come at it as web development with IO you are going to have a very bad time.
This! Simple schedulers generally only allow system calls (such as printf) from the main thread. If you really want to 'print' from a child thread then send a message to the main thread, asking that it prints the message contents on on behalf of the child thread.
The USB CDC version is much faster but may not always be appropriate for your project.
I implemented one that uses a buffer and an interrupt to feed the UART in the background, and then I disabled the plugin the Pico SDK provided. I'm not using an RTOS.
The SDK example [uart_advanced](https://github.com/raspberrypi/pico-examples/blob/master/uar...) [github.com/raspberrypi] shows how to set up UART interrupts using the Pico SDK
I went ahead and rolled a simple green thread timer.
It doesn't support actual process management like a real kernel, and doesn't make any guarantees about anything, but it has gotten me further than bare metal scheduling and helped me avoid the shit show that is RTOSes.
Think JavaScript timer callbacks with an optional context struct (in C)
It has allowed me to interrogate a variety of sensors, process inbound signals, make control decisions and emit commands, all at various frequencies.
I highly recommend folks try something like this before ruining your life with these slow, abstract architectures.
I’d also like to suggest uC/OS. It doesn’t take much more than setting a timer in asm to port that rtos but I haven’t had the time myself to try it.
Its might be the most used ThreadX on the market in the professional space (due to its history), but its quite frankly stupid simple to use.
It was initially developed by Express Logic, then partnered tightly with Renesas, Then sold to Microsoft, and then transferred to the Eclipse foundation for good.
They also provide NetX(duo), FileX, LevelX, UsbX and GuiX with full source and the same licensing afaik. Personally I don't care for UsbX or GuiX.
It would likely not be a bad option.
ARMv7 is the minimum, and MMU is required. RISC-V is recommended.