Ask HN: How to get started with real-time Linux? | Hacker News Reader

Ask HN: How to get started with real-time Linux?

67 pointswithoutsnow49 comments

I would like to learn and work on both software and hardware of real-time Linux. Any books or projects do you recommand for getting started?

View on HN

I found this to be hard-ish to get started as well. However, it turns out you just need to write your program to be performant, and know a few tricks to tune the OS and the scheduler. After looking on the internet and failing to find material about this, I wrote a series of blog post for this topic recently, starting with this post: https://shuhaowu.com/blog/2022/01-linux-rt-appdev-part1.html

The series is mostly complete, except one to two posts on more advanced topics like validation and lockless programming.

Real-time software does not mean fast software! It mean determinism, guarantee that each time it run it will never take more than x milliseconds to complete. Those guarantee are very hard when considering a system like linux with many background process or thread each randomly doing IO and allocating memory.

That depends on whether we're talking about hard-realtime or soft-realtime.

[0] https://www.google.com/search?q=hard-realtime+soft-realtime

> A hard-real time system is a system in which a failure to meet even a single deadline may lead to complete or appalling system failure. A soft real-time system is a system in which one or more failures to meet the deadline are not considered complete system failure, but that performance is considered to be degraded.

even in the case of soft-realtime we don't say the system is real-time because it's fast. We say its realtime because majority of the time it meet its deadline.

Yes, at the same time I don't know if I'd characterize a soft-realtime system as slow. One way to meet deadlines is by being very efficient/fast, while keeping variance low (e.g. no stop the world gc).

By that definition, a phone call that always takes 10 seconds for one side to hear the other is realtime.

That would indeed be a realtime system by common definitions as long as that latency is guaranteed. Audio and video are commonly considered soft-realtime domains for exactly that reason.

Yep. If you mean low-latency, say that. Real-time is an unfortunate name, but it means what it means.

And phone … callthatju arousmpnnd is n’t realtime.

Technically not relevant but I know that a lot of early handhelds that repurposed RTOS for GUI felt slow, jumpy and unresponsive, perhaps because everything was polled? In that sense RTOS can be slow.

I think it felt that way because of bad design + slow hardware. You see the exact same thing by installing a modern windows version or some desktop like GNOME to low-end computers.

Who's that "we" ? For anyone I've worked with, real-time also implied "reaction time is faster than human perception"

You're right, but I think it's mostly semantic and depends on how you define "performant" or "fast". Real-time software is about being "fast enough", every time. After all, slow and fast is subjective measures that varies wildly depending on the context.

In the context I'm working with (1000Hz hard real-time), the software has to be "performant" (running algorithms like collision detection) in a deterministic manner as each loop iterate has less than 800us of time. When developing such a system, the developer needs to accurately know the performance characteristic of the software ahead of time.

In another system, fast might mean 5ms, which is unacceptably slow in the context above.

It's not Linux but if you wanted to learn about real time, you could check out Ardupilot for drones. You'll get to learn about a variety of topics including:

- hardware (embedded hardware, brushless motors, propellers, batteries, radio, etc.)

- guidance (sensor fusion algorithms, kalman filtering, etc.)

- navigation (GPS/GNSS, magnetometers, barometers, accelerometers, gyroscopes, etc.)

- control (PWM control, PIDs, feed forward loops, etc.)

While drones/quadcopters are a more continuous physical system than any clocked digital circuit, they can be controlled quite well with algorithms updating at "only" 1khz, which is probably a relatively wide open door for someone to gatekeep "that isn't realtime"

At least in the usage I've seen realtime refers to guarantees on latency and interrupt timers, not necessarily the amount of latency or polling rate. A system that polls at 1mhz but might miss a poll is not realtime, while a system that polls at 1hz with hard deadline guarantees would be considered realtime.

Hi!

Is there a good training ground, like a simulation, for playing around with this stuff?

There's robotics simulators like Gazebo you could try.

Your best starting point in LinuxCNC it may very well be the single greatest application for real time linux

Real-Time is a broad field and implemented in one of 3-ways a) On bare-metal, b) Using a RTOS (eg. FreeRTOS, QNX) c) On Linux/etc. As such you have to understand what is actually involved when you say something is "real-time" (Hard/Soft). As a first approximation "real-time" can be said to be about minimizing latency (within accepted bounds) of service requests (ISRs, task functions etc). Thus if you do it via (a) above you get the best control but have to do everything, (b) gives you some framework support via a RTOS kernel and (c) is always "soft real-time" unless you break out of the kernel. The difficulty increases as you go from (a)->(b)->(c) since there are more intermediate layers and abstractions. But this is the best way to approach the study of the subject. With that in mind i can suggest the following books;

1) Real-Time Embedded Systems: Design Principles and Engineering Practices by Xiaocong Fan. - Good text book.

2) Introduction to Embedded Systems: Using Microcontrollers and the MSP430 by Manuel Jiminez et.al. - Very good book teaching you bare-metal MCU programming.

3) Hands-On RTOS with Microcontrollers: Building real-time embedded systems using FreeRTOS, STM32 MCUs, and SEGGER debug tools by Brian Amos - Great book showing the use of all tools/ecosystem needed for RTOS programming.

4) POSIX.4 Programmers Guide: Programming for the Real World by Bill Gallmeister. - The classic indispensable work on "real-time programming" for Unix-like systems.

The #4 reference (gallmeister) is getting dated. The tarball of example source is still out there on the web.

At the practical level, I would suggest the Redhat Realtime linux tuning guide. It covers practical ways to identify and remove latency and jitter from your system.

My experience with the RH realtime kernel is 120 hz is usable. 60 hz can be very good. Of course, YMMV....

IMO, Gallmeister is still the best beginner's book since it explains the POSIX real-time profile apis succinctly. VxWorks/QNX/etc. all implement most of these apis thus giving the programmer a single view of the system which can be carried across most OSes.

Another very good book i forgot to mention is:

5) Real-Time Systems and Programming Languages: Ada, Real-Time Java and C/Real-Time POSIX by Alan Burns and Andy Wellings.

>At the practical level, I would suggest the Redhat Realtime linux tuning guide. It covers practical ways to identify and remove latency and jitter from your system.

Agreed, but it deals with configuring the system rather than explicit coding. I listed it in another comment in this thread - https://access.redhat.com/documentation/en-us/red_hat_enterp...

Real-time linux can be thought of taking vanilla linux - configuring the kernel in a certain way, booting your kernel with certain boot params and writing your real-time application in a certain way.

While there quite a few hiccups along the way - you will be able to bring down jitter or OS noise to around 1 us. Of course a microkernel based OS like QNX can take you down to around 1 ns - but then the point of using Linux is that you are able to utilize the ecosystem around it.

Look into isolation, real-time patch and high performance computing.

If it must not be linux, I would recommend going for something embedded, like FreeRTOS or similar. The advantages: - Linux + Real Time has become a buzz word, and you will find lots of bad information - If you want to learn HW, nothing better as embedded, where you interact with the hardware. - The FreeRTOS is so small you can really understand it pretty fast. You can see how it is made, much easier than Linux. - The essentials you learn with that, will work with Linux later.

I don't know that I'd recommend learning FreeRTOS for fun because you should basically never be using a traditional RTOS if you can possibly avoid it. Unless you really like reinventing stuff that you get "for free" on Linux, there are remarkably few advantages beyond certifiability (which FreeRTOS in particular doesn't get you), easier porting (just buy something that actually runs Linux) and lower resource use (more work to actually achieve!).

As for Linux, your first stop should be the bootlin materials [1]. They're fairly comprehensive and regularly updated.

[1] https://bootlin.com/docs/

While I have not had a chance to work with it yet, there is a path to cert from FreeRTOS in SAFERTOS

https://www.highintegritysystems.com/safertos/

I spent a fair bit of time at my last job trying to convince people that migrating to SAFERTOS and certifying from there was easier than certifying from FreeRTOS, to no avail. It's more like a fork than a drop-in replacement once you're far along in a project, but it's certainly the better path to take than the alternative. I'd still say the developer and EE experience on Linux is far better than either though.

You are aware that real time only means guaranteed right?

What is your goal as this is a broad field?

If you wanna just start with it, I would try to read up on cars, can Bus and how all of that works.

Is Linux a popular option for real-time? Or are other OS more relevant in this field?

FreeRTOS is a better choice in a lot of circumstances. But you don't get a driver ecosystem like Linux.

In embedded the trend is heading toward heterogenous systems, where you have an application processor running Linux and a coprocessor (sometimes on the same die) that runs an RTOS.

Zephyr is taking over that space since it does drivers in a similar fashion to Linux. https://www.zephyrproject.org/

According to this trend, what actually runs on Linux and what runs on RTOS?

In some of the systems I've worked with the Linux side runs the GUI, mass storage, connectivity, etc while the small RTOS processor manages the time-critical control and measurement.

Depends how "hard" realtime we're talking. For tighter deadlines you'd usually use QNX or PikeOS or the like. The bigger issue is that for higher task frequencies you probably don't want to use big microprocessors, but microcontrollers that don't have a MMU but a MPU max. That rules out Linux quite often (plus space constraints).

The best I've managed to achieve is by taking Linux off the real-time equation as much as possible : use isolcpu to tell Linux to not schedule processes on specific cores, put them there yourself with pthread API, nice etc. You can also control on which core interrupts happen fairly easily. This gives you almost microcontroller-like access to some of your CPU cores.

Neat! Would be nice if you could add some more details to each of the points mentioned. As an example;

1) Do i need to configure "isolcpus" if i use "pthread_setaffinity" in my code directly ?

2) How do you do "interrupt affinity" in code ?

PS: I found Red Hat's documentation for "real-time" useful: https://access.redhat.com/documentation/en-us/red_hat_enterp...

> 1) Do i need to configure "isolcpus" if i use "pthread_setaffinity" in my code directly ?

yes: isolcpus is there to tell the kernel : "never schedule anything on this CPU" ; pthread_setaffinity is then there to move your real-time process/thread on the now-isolated CPU. Maybe if you control the entire user space you could get the same result by manually calling pthread_setaffinity for every single thread, but that's very likley not what you want (and I'd guess the kernel would still do some periodic polling in this case which they won't on isolated cpus).

But most the information I have comes from reading this exact guide which is very, very good and comprehensive :)

I think I have been on a similar path of thought: While there are many speed tricks and parallelized processing to try and to use in modern PC, there are surprisingly little available resources for realtime, as in synchronous-to-wall-clock, processing.

You can set up a timer interrupts in Arduino, There are web-JavaScript tutorials to set up callback function to execute every 3000ms, but you won't be doing RPC over network with timestamps and expiration, without going neck deep in something specialized and chaotic as ROS2.

I'm picturing some sort of distributed, networked, timestamped system that can schedule remote executions and construct time correlated observations, something obvious to designers/architects of original SNMP spec and NIST 4D/RCS project, and I'm predicting we'll have an industry standard framework just casually appearing by the end of this decade. It's weird we don't have one.

Basically:

- Traditional Unix optimises for sharing resources and using them efficiently - 'multiprocessing', 'time slicing', 'resource sharing', 'garbage collecting', etc.

- Realtime: optimises for deterministic low latency. SCHED_FIFO processes that use a logical CPU and do not yield during the trading day, use money to buy more memory or crash rather than allow the GC to destroy your latency during the trading day.

https://access.redhat.com/documentation/en-us/red_hat_enterp... is RHEL RT focused but the kernel optimisations discussed are all OSS licensed.

Don't.

Linux is too complex to provide any sort of guarantee; real-time Linux is just soft realtime.

Look at an actual RTOS instead, such as seL4[0].

0. https://sel4.systems/About/seL4-whitepaper.pdf

This is awesome, I hadn't seen this before. How would you compare developing on this with something like FreeRTOS or some other RTOS that doesn't really have a user space? I get the security benefits, I'm curios about the developer experience, how much code you have to write yourself to get going, how's the toolchain, etc.

I seem to recall an ubuntu music variant distro with the RT patch baked in but cannot for the life of me dig it up.

This however may give you a nice point to dive in from:

https://ubuntu.com/engage/an-introduction-to-real-time-linux...

for more advanced projects consider getting preempt-rt going in a yocto install.

Ubuntu Studio. They removed the rt kernel a while ago though. From some googling it seems they now use a "Low Latency Kernel" which users say has lower actually latency than the rt kernel at the expense of no hard guarantees on the latency.

Apparently a malicious or poorly coded rt process could also completely lock the machine on rt kernel.

https://help.ubuntu.com/community/UbuntuStudio/RealTimeKerne...

> Apparently a malicious or poorly coded rt process could also completely lock the machine on rt kernel.

This is false by default. The kernel reserves 5% (configurable) of CPU cycles for non-RT tasks, meaning that no RT task can lock the machine unless it also blocks all IRQs; this could be done by a non-RT task too, so it is not an RT-related hazard.

See

  /proc/sys/kernel/sched_rt_period_us
  /proc/sys/kernel/sched_rt_runtime_us

It is true that if you misconfigure these values, things can go wrong, but that's true of a lot of kernel parameters.

Others are offering good books and things, but my experience during my autonomous vehicle days, I found just seeing how rt-linux was being used by others during market research was very useful in understanding how it was being used and where it failed and needed a more tested RTOS. So my suggestion is start searching the startup space and see how its being used these days.

Not a book or a project, but I can't recommend enough using Xenomai4/EVL. Under the common use case of "some small kernel of code needs bounded latency" plus "I want to use normal Linux and not think about realtime shenanigans damnit," you'll have a substantially easier time developing in that environment than anything else I'm aware of.

Having only dealt with this at the level of writing code for a RISC microcontroller, another facet I don't see mentioned that may be of importance is the following CPU implementation detail.

Code compiles to assembly instructions which typically appear atomic to the author, however, such instructions may take variable numbers of cycles to execute.

1. install hwlatdetect

2. use hwlatdetect

3. give up when you realize you can't buy commodity hardware with documented NMIs or open-source firmware

I am not an expert on the matter but as far as I understand you need the appropriate hardware platform, too? Last time I did some research about it I came to the conclusion that for instance Raspberries make for a poor RT platform. Is that not the case?

https://rt.wiki.kernel.org https://en.m.wikipedia.org/wiki/RTLinux

Reply on news.ycombinator.com