RTEMS Real Time Operating System
rtems.org
rtems.org
Today the "MS" stands for "Multiprocessor Systems". Heterogenous multiple processors [1]. The OS is perfectly happy running x86, ARM and RISC-V in the same system, though I don't know which of the canned Board Support Packages use that capability.
It's a pretty good introduction to RT--small enough to get your head around, but lots of capability. I used it for my first major embedded software project (LAN NIC driver, based on Linux driver for similar chip) 15 years ago and it was pretty straight-forward. Rate it 4 1/2 stars, would code again.
[0] https://devel.rtems.org/wiki/History/Timeline [1] https://docs.rtems.org/releases/rtemsdocs-4.8.0/share/rtems/...
Did find a performance bug in the core I don't remember much about (unfortunate choice of operations that flushed caches or something and realllly slowed down... something). Submitted the half-a-dozen line patch upstream that Joel Sherrill pulled without drama, even though my test case was overly complicated and opaque.
What we needed was something like Linux's scheduling queue disciplines [0]; it would have allowed us to say "here's a packet, schedule it for 10 mSec from now", and we'd be good as long as the vast majority of packets took less than 10 msec to wend through the stack. But that didn't exist at that point.
Because RTEMS is open-source and also because its internal structure is many orders of magnitude easier to understand than that of something like Linux, or even of some *BSD, extreme customization can be done with relatively modest work.
When you do not need them, it is easy to remove completely the implementations of many parts of the API and of many classes of kernel objects, to obtain a much leaner RTOS kernel.
Any causes of jitter from the kernel or device drivers (if they occur in higher priority tasks, as the lower priority tasks should not be able to interfere) should be relatively easy to identify and patch.
I have used once RTEMS for an application with many channels of synchronous multiplexed telephony, with very tough jitter requirements, and there have been no problems, but it was not RTEMS out-of-the-box, it was customized for the application, i.e. with about half of the kernel removed, a new interrupt handler added, and with custom device drivers.
I had to port it to PowerPC, because that was an architecture not yet supported at that time, but the porting was very easy, because the code quality and the documentation were excellent and all parts dependent on the CPU architecture were clearly separated from the rest. I also had to write some device drivers, including an Ethernet driver, and again, the documentation and the device driver examples made that task easy.
RTEMS proved to be a much better choice than the alternatives at that time, i.e. pSOS and VxWorks, because the access to the source code enabled a degree of customization impossible with the proprietary RTOSes and its performance was great too.
In later years, I have not used RTEMS much, but not because I have discovered any better RTOS, but because I have realized that for a majority of the embedded software using an RTOS is an overkill. It is simpler and with less overhead to just use on bare metal an event handler with a priority queue storing scheduled actions whose time duration is limited (this is equivalent with cooperative multitasking).
There are applications which can be simplified by the preemption offered by an RTOS, like RTEMS (typically when porting to an embedded device some programs that have been written for a general-purpose OS, e.g. for Linux or FreeBSD), but the need for this must be assessed carefully, because using preemption guarantees that you cannot use the CPU close to its full capacity.
Therefore, using RTEMS (or any RTOS, but RTEMS is one of the best) can be the best choice for an embedded system when the CPU is quite powerful in comparison with what it has to do, so optimizing the CPU utilization is not needed, and the main target is to minimize the software development time by reusing some programs which have not been written specifically for real-time systems (e.g. a Web server or other complex networking programs).
Without an RTOS, i.e. without preemption, it is mandatory to be able to estimate upper limits for the execution time of any function, to be able to guarantee that deadlines are met. With an RTOS, you must still be able to do that for functions that are executed by high-priority tasks, but this may be unnecessary for those executed in low-priority tasks (which will be preempted, whenever necessary, by the high-priority tasks).
I remember the RTEMS event concept was not very good- the events are globally shared. You are better off using semaphores.
>I have realized that for a majority of the embedded software using an RTOS is an overkill
I agree with this. Instead of an OS, we like to use a small framework that provides a CLI, database and UART communications (YMODEM protocol). It's a single stack system that provides a simple work-queue for scheduling callbacks, and for interrupt code to schedule non-interrupt code. It works for almost everything as long as you are comfortable programming in an event-driven style.
https://github.com/nklabs/libnklabs
(recently added support for AVR 8-bit processors).
Like most of ESA's software engineering tools, it is hopelessly outdated and relies on a patched GCC 4.2.1 which is missing a lot of useful features and bug fixes, especially for the SPARC architecture which is often used in ESA spacecraft. See -mfix-* switches here https://gcc.gnu.org/onlinedocs/gcc-10.4.0/gcc/SPARC-Options....
While RTEMS is GPL-licensed ESA does not want the code to be freely available -- you probably won't find it online. Some years ago they realized that, if they don't upstream their code, they have no chance to keep up to date and started an initiative to improve the situation: https://rtems-qual.io.esa.int, https://rtems-qual.io.esa.int/
Nothing against you personally, but I would really appreciate if instead of stating a certain believe, you actually look it up and point out a source for that bit of information.
* SpaceWire support (LEON3)
* MIL-STD-1553 support (LEON3)
* kernel and application monitoring services (whatever that means)
Do you know any other differences?
Quietly an RTEMS core developer submitted GCC support for running gcov or gprof on embedded targets and getting the information off. This will be used to get coverage reports which meet ESA expectations. We do have very high code coverage.
ESA has also sponsored formal analysis of some parts of RTEMS.
I am quite happy ESA is working so well with our core developers and community this time.
https://docs.rtems.org/branches/master/user/bsps/bsps-arm.ht...
> The ‘raspberrypi’ BSP supports the single core models (Zero, Zero W, A+, B+), and the ‘raspberrypi2’ BSP supports the Raspberry Pi 2, Raspberry Pi 3 A+, and Raspberry Pi 3. The Raspberry Pi 4 is not supported currently. The default bootloader on the Raspberry Pi which is used to boot Raspbian or other OS can be also used to boot RTEMS. U-boot can also be used.
Being naive/curious is the best way to start learning.
https://en.wikipedia.org/wiki/Real-time_computing#Criteria_f...
>>In the context of multitasking systems the scheduling policy is normally priority driven (pre-emptive schedulers). In some situations, these can guarantee hard real-time performance (for instance if the set of tasks and their priorities is known in advance). There are other hard real-time schedulers such as rate-monotonic which is not common in general-purpose systems, as it requires additional information in order to schedule a task: namely a bound or worst-case estimate for how long the task must execute. Specific algorithms for scheduling such hard real-time tasks exist, such as earliest deadline first, which, ignoring the overhead of context switching, is sufficient for system loads of less than 100%.[7] New overlay scheduling systems, such as an adaptive partition scheduler assist in managing large systems with a mixture of hard real-time and non real-time applications.
Normal operating systems provide time slices in big chunks, and not very predictably.
The RTOS approach is less efficient overall and has lower total throughput, but some applications don’t need a lot of horsepower, just consistency.
Think: robotics, audio, servo motor controllers, etc…
The general-purpose operating systems attempt to execute all available tasks as quickly as possible and in a fair manner between users, but they cannot make guarantees about the worst-case execution times, which can be very bad even when the average execution times are very good.
In order to be able to provide time guarantees, a RTOS avoids any algorithms that are not deterministic and it uses different policies for scheduling the execution threads, unlike those used by a general-purpose OS.
Most RTOSes use fixed priorities for the execution threads, where any higher-priority thread always preempts immediately a lower-priority thread, whenever an interrupt signals an event which must be handled by the high-priority thread, while the threads of equal priority are scheduled in a round-robin manner.
Unlike when programming for a general-purpose OS, when programming for a RTOS a very important additional task for the programmer is to assign priorities for all the execution threads. If the priorities are assigned in a wrong way, it can be impossible to guarantee the time deadlines.
For some common cases there are easy rules for priority assignments. E.g. when having a set of threads that are activated periodically, which must always finish their work within a period, then the priorities must be assigned, from the highest to the lowest, to the threads ordered from the smallest period to the greatest period.
The programs which may execute for an indefinite time after they receive control, like many of those written for Windows or Linux, must be put on the lowest priority level available, because any program with a lower priority than them would never be executed by an RTOS. When there are more such programs, they are executed in round-robin manner, being preempted after a configurable time interval.
The old brochure is pretty interesting overview of the program goals.
Did you know that we provide a tool to build all tooling from source including application of patches? It's similar in concept to BSD ports. This gives users full source and the ability to put it under configuration control for long term projects.
Several decades ago, when I was young and I have discovered RTEMS by chance, I have learned many useful things from its documentation, source code and tooling scripts. IIRC, I have also got some very helpful e-mail responses from you, to some of my newbie questions.
So I have always been grateful for your RTEMS work.
One of the useful things that I have learned then from the RTEMS scripts was precisely how to build my own customized toolset with binutils/gcc/gdb and so on, for cross-development for an embedded system target.
I have always done this, instead of attempting to use one of the pre-built but less suited for my purposes toolsets that can be found in various places, and it has always helped me a lot.
I'd written off RTEMS as a serious modern RTOS, but it looks pretty alive. I am intrigued by the FreeBSD heritage of the network code. Anyone tried both?
We also have a port of lwip which is appropriate for smaller systems.
Finally we have the older legacy FreeBSD stack which we don't recommend for new projects but some BSPs still only have drivers in there.
xtensa/tensilica is a good architecture, cheap and lean, unfortunately its biggest problem seems to be its harvard architecture and weird memory structure because out there I find very few loader/relocator impementations (nominally I only know niicoooo's version[1], brian pug's derivative[2] for his interesting joltos, mongoose/Zephyr's supercustom versions, and my own) and this seems to be the main stumbling block for new os development in this platform
[1] https://github.com/niicoooo/esp32-elfloader
[2] https://github.com/joltwallet/jolt_wallet/tree/master/jolt_o...
Even with external memory, I'm not sure if I've seen any ESP32 with more than 8MB of SPI-attached RAM - this would be enough for RTEMS, but I'm not sure about viability of using SPI-attached SRAM as main memory.
EDIT: Looking through processor supplements documentation for RTEMS, the 520kB of RAM in ESP32 would be enough. I guess RTEMS is simply less well known and its main users target heavier platforms whereas many people are fine with just the provided ESP32 toolkit as base "OS"
Pedantry: everyone uses a Harvard architecture, that just means having separate buses internally for instructions and data -- essentially if you have a separate icache/dcache, you're Harvard. Xtensa is weird because it's L1 caches are incoherent.
As far as the architecture, it's sort of too clever for its own good. The register windowing scheme seems amazing, just a few comparators at runtime and some special exception handlers is all it takes to get 64 hardware registers to work. But... no one ever taught compilers how to deal with the complexity (the generated code needs to decide how many of its local registers to preserve across a call, which is a global optimization that depends on the behavior of the code it's invoking). So the end result is that for the same code, Xtensa code actually uses significantly more stack space than conventional architectures like ARM or RISC-V. (Their own toolchain is better at this than gcc, but still bad).
And FWIW: Zephyr doesn't do dynamic loading yet. We have crazy linker behavior, but it's all static.
A popular chip without a port hints that they have not found it a good choice for building their extremely reliable systems.
Not everything has to be especially reliable. Cheap is a merit of its own, as is, often, newness. Targets with maintained ports are often slower, less capable, less kitted out than what catches your eye. But if it has been out for some time without a port, it is very possible it is because it was tried and rejected for reasons. You are not obliged to find their reasons persuasive, but it might be wise to find out what they are.