The real realtime preemption end game
lwn.net
lwn.net
The QNX kernel doesn't do anything with strings. No parsing, no formatting, no messages.
Linux suffers from being too bloated for real time. Millions of lines of kernel, all of which have to be made preemptable. It's the wrong architecture for real time. So it took two decades to try to fix this.
" We took a substantially different approach with seL4; its model for managing kernel memory is seL4’s main contribution to OS design. Motivated by the desire to reason about resource usage and isolation, we subject all kernel memory to authority conveyed by capabili- ties (except for the fixed amount used by the kernel to boot up, including its strictly bounded stack). "
I guess I should've said seL4 took capabilities to the extreme.
L4's contribution was speed. It was assumed that microkernels, and especially capability-based microkernels were fundamentally slower than monolithic kernels. This is why Linux (1991) is monolithic. Yet L4 (1994) was the fastest operating system in existence at the time, despite being a microkernel and capability based. It's too bad those dates aren't reversed, or we might have had a fast, capability-based, microkernel Linux :(
The problem with shutdowns and restarts is the secure bootstrapping problem. The boot process must be within the trusted computing base, so how do you minimize the chance of introducing vulnerabilities? With checkpointing, if you start in a secure state, you're guaranteed to have a secure state after a reboot. This is not the case with other any other form of reboot, particularly ones that are highly configurable and so easy for the user to introduce an insecure configuration.
In any case, many apps are now designed to restore their state on restart, so they are effectively checkpointing themselves, so there's clearly value to checkpointing. In systems with OS-provided checkpointing it's a central shared service and doesn't have to be replicated in every program. That's a significant reduction in overall system code that can go wrong.
That's not true actually. These capability systems have the principle of least privilege right down to their core. The checkpointing code is in the kernel which only calls out to the disk driver in user space. The checkpointing code itself is basically just "flush these cached pages to their corresponding locations on disk, then update a boot sector pointer to the new checkpoint", and booting a system is "read these pages pointed to by this disk pointer sequentially into memory and resume".
The attack surface in this system is incomparably small compared to the boot process of a typical OS, which run user-defined scripts and scripts written by completely unknown people from software you downloaded from the internet, often with root or other broad sets of privileges.
I really don't think you can appreciate how this system works without digging into it a little. EROS was built from the design of KeyKOS that ran transactional bank systems back in the 80s. KeyKOS pioneered this kind of checkpointing system, so it saw real industry use in secure systems for years. I recommend at least reading an overview:
https://flint.cs.yale.edu/cs428/doc/eros-ieee.pdf
EROS is kind of like what you'd get it if you took Smalltalk and tried to push it into the hardware as an operating system, while removing all sources of ambient authority. It lives on as CapROS:
Oh, and anytime as needed the OS could serialize out your process, pipe it across the network to another machine, and resume. Useful for load balancing, or relocating a program to be near the data it is accessing. Unless your program pays special attention to the clock, it wouldn't notice.
I still think about Amoeba from time to time, and imagine what could have been if we had gone down that route instead.
E.g. You spawn a process on another computer and then the connection drops.
In practice it might not be an issue. The reason you'd typically do something like move processes across a WAN is because you want it to operate next to data it is making heavy use of. The copy that booted up local to the data would continue operating, while the copy at the point of origin would suddenly see the data source go offline.
Now of course more complex schemes can be devised, like if the data source is replicated and so both copies continue operating. Maybe a metric could be devised for detecting these instances when the partition is healed, and one or both processes are suspended for manual resolution? Or maybe programs just have to be written with the expectation that their capabilities might suddenly become invalid at any time, because the capability sides with the partition that includes the resource? Or maybe go down the route of making the entire system transactional, so that partition healing can occur, and only throw away transaction deltas once receipts are received for all nodes ratcheting state forward?
It'd be an interesting research area for sure.
seL4 is now a healthy non-profit, seL4 foundation[1].
0. https://microkerneldude.org/2022/02/17/a-story-of-betrayal-c...
1. https://microkerneldude.org/2022/03/22/ts-in-2022-were-back/
With that out of the way, you're right in that the microkernel doesn't present a posix interface.
But, like QNX, there are libraries for that, seL4 foundation itself maintains some.
They have a major ongoing effort on system servers, driver APIs and ways to deploy system scenarios. Some of them were talked about in a recent seL4 conference.
And then there's third party efforts like the amazing Genode[0], which supports dynamic scenarios with the same drivers and userspace binaries across multiple microkernels.
They even have a modern webbrowser, 3d acceleration as well as providing a virtualbox box that runs inside Genode, so the dogfooding developers are be able to run e.g. Linux inside a virtualbox to bridge the gap.
I'm not bothered by the kernel bloat. There's a lot of dev time being invested in Linux and while the desktop is not as much of a priority as say the server space a performant kernel on handhelds and other such devices and the dev work to get it there will benefit the desktop users like myself.
A bunch of similar embedded systems. And blackberry, if anyone's still using them.
In the end, all I could practically use it for was as a bluetooth audio connector.
So much like a well structured main method in a C program or other C like language where main just orchestrates the calling of other functions and such. In this case main might initialize different things where the QNX kernel doesn't but the idea or general concept remains.
I'm no kernel dev but this sounds good to me. Keeps things simple.
The process of serving an HTTP request is broken into a large number of fine grained stages and plugin modules may hook into any or all of these to modify the input and output to each stage.
The same basic idea makes it easy to turn any application concept into a modules-and-core architecture. From the day I read (skimmed) that book a decade or two ago this pattern has been burned into my brain
Pretty much everywhere there is some undercurrent of "use this ultra-small generic interface for everything and life will be easier". With games and ECS, microkernels and IPC-for-everything, with frontend frameworks and components that only communicate between themselves via props and events, with event sourcing and CQRS backends, Actors in Erlang, with microservices only communicating via the network to enforce encapsulation... Perhaps even Haskell's functional-core-imperative-shell could count as that?
I feel like OOP _tried_ to get to this point, with dependency injection and interface segregation, but didn't quite get there due to bad ergonomics, verbosity and because it was still too easy to break the rules. But it was definitely an attempt at improving things.
~90% of those are device drivers, which you'd still need with a microkernel if you want it to run or arbitrary hardware.
The only downside of course, is that you don't get the isolation benefits of the microkernel for anything depending on the Linux kernel process.
https://netflixtechblog.com/kubernetes-and-kernel-panics-ed6...
I understand it's less elegant, efficient, etc. But sometimes commodity wins over correctness.
For automotive and aerospace, you really want the control systems to be able to run no matter what.
I'm unclear on why you put "many cycles to spare for misses" in quotes, as if it's unimportant. If a linux/arm (or x86) solution is displacing a much lower speed "real real time" solution, that's the situation...the extra cycles mean you can tolerate some misses while still being as granular as what you're replacing. Not for every use case, but for many.
Maybe they used the quotes because they were quoting you, haha.
As for the quotes, it was a direct citation, not a way to dismiss what you said.
Actual realtime systems are about workload scheduling that takes into account processing deadlines. Hard realtime systems can make guarantees about processing latencies, and can preemptively kill or skip tasks if the result would arrive too late. But this is not something that the Linux kernel can provide, because it is a system property rather than about just the kernel: you can't provide any hard guarantees if you have no time bounds for your data processing workload. So any discussion about -rt in the context of the Linux kernel will always be about soft realtime only.
In the case of the kernel, it is indeed too large to be considered hard realtime. Best case we can make it into a firmer realtime than it currently is. But I would place it nowhere near avionics flight calculators (like fly-by-wire systems).
For example, if I have a video game where there's collision against the walls, I can understand this as potentially colliding against "multiple things simultaneously", since I'm likely to describe the scene as a composite of bounding boxes, polygons, etc.
But to get an answer for what to do in response when I contact a wall, I have to come up with an algorithm that tests all the relevant shapes or volumes.
The concurrency bug that appears when doing this in a naive way is that I test one, give an answer to that, then modify the answer when testing the others. That can lead to losing information and "popping through" a wall. And the direction in which I pop through depends on which one is tested first.
The conventional gamedev solution to that is to define down the solution set so that it no longer matters which order I test the walls in: with axis aligned boxes, I can say "move only the X axis first, then move only the Y axis". Now there is a fixed order, and a built-in bias to favor one or the other axis. But this is enough for the gameplay of your average platforming game.
The generalization on that is to describe it as a constraint optimization problem: there are some number of potential solutions, and they can be ranked relative to the "unimpeded movement" heuristic, which is usually desirable when clipping around walls. That solution set is then filtered down through the collision tests, and the top ranked one becomes the answer for that timestep.
Problems of this nature come up with resource allocation, scheduling, etc. Some kind of coordinating mechanism is needed, and OS kernels tend to shoulder a lot of the burden for this.
It's different from real-time in that real-time is a specification of what kind of performance constraint you are solving for, vs allowing any kind of performance outcome that returns acceptable concurrent answers.
I bet there are hard-realtime (commercial) OSes running on ARM, and the ability to use a lower-specced (cheaper, simpler, consuming less power) core may be seen as an advantage enough to pay for the OS license.
The power issue is real, but it might well be the same price or cheaper - a standard ARM that gets stamped out by the million can cost less than a "simpler" microcontroller with a smaller production run.
Any system that deadlocks.
Perhaps even that's not an example of such a system, 0.3 microseconds is close to the allowable real-time budget, and QC would probably not scrap a $20k part if you were off by that much once.
But in practice, every time I've heard "soft real time" suggested, the failure mode is not a sub-microsecond miss but a 100 millisecond plus deadlock, where a hardware watchdog would be needed to drop the whole system offline and probably crash the tool (hopefully fusing at the tool instead of destroying spindle bearings, axis ball screws, or motors and gearboxes) and scrap the part.
I’m trying to understand where the roadblock on a rPi + small FPGA hybrid board for $50 fails at the task… and it sounds like the OS/firmware doesn’t suffice. (Or a SoC, like a Zynq.)
Eg, if we could guarantee that the 1.5GHz core won’t “be off” by more than 1us on responding and the FPGA can manage IO directly to buffer out (some of) the jitter, then the cost of many hobby systems with “(still not quite) hard” real time systems would come down to reasonable.
Hobby systems like a small CNC mill or lathe usually don't need anything near 1µs (or better) max jitter. LinuxCNC (derived from NIST's Enhanced Machine Controller, name changed due to legal threats) runs fine on preempt-rt with control loops around 1kHz, with some systems you can also run a "fast" thread with say 20kHz and more to generate stepper motor signals, but that job is best left for the FPGA or an additional µC IMHO.
But it's not really so much about being fast, it's about being able to guarantee that your system can respond to an event within a given amount of time every time. (At least that is how a friend who works in embedded/real time explained it to me.)
If you really need realtime, then you really need it and "close enough" doesn't really exist.
This is just my perception as an outsider though.
Even if a system can run (RT-)Linux, it doesn’t mean it’s suitable for real-time. Hardware for real-time projects needs much lower interrupt latency than a lot of hardware provides. Preemption isn’t the only thing necessary to support real-time requirements.
linux RTOS is at microseconds granularity but it still can not 100% guarantee it, anything in cache nature (L2 cache, TLB miss) are hard for hard real time.
a dual kernel with xenomai could improve it, but it is not widely used somehow, only used in industrial controls I think.
linux RT is great for audio, multimedia etc as well, where real-time is crucial, but not a MUST.
yup that's why you'd pin the memory and the core for the critical task. which, alas, will affect performance of the other cores and all other tasks. and whoosh there goes the BOM...
which again as we both probably are familiar with leads to the SoC designs with a real time core microcontroller and a HPC microprocessor on the same package. which leads to the question how to architect the combined system of real-time microcontroller and compute power but soft real time microprocessor such that the overall system remains sufficiently reliable...
oh joy and fun!
How the CPU and MCU communicate is a good question to tackle, typically chip vendors provide some solutions, I think OpenAMP is for this.
Linux-rt makes linux actually decent at soft realtime. PREEMPT_RT usually results on measured peak latency for realtime tasks (SCHED_RR/SCHED_FIFO) on the order of a few hundred usec.
Standard Linux lets latency go to tens of milliseconds, easily verifiable by running cyclictest from rt-tests for a few hours while using the computer. Needless to say, this is unacceptable for many user cases, including pro audio, videoconference and even gaming.
In contrast, AmigaOS's exec.library had no trouble yielding solid sub-millisecond behaviour in 1985, on a relatively slow 7MHz 68000.
No amount of patching Linux can give you hard realtime, as it is about hard guarantees, backed up by proofs built from formal verification, which Linux is excluded from due to its sheer size.
There's a few RTOSs that are formally verified, but I only know one that provides process isolation via the usual supervisor vs user CPU modes virtualization model: seL4.
https://www.audioadvice.com/audioquest-nrg-dragon-high-curre...
(I didn't know the term, trying to be helpful if others don't)
Depending on implementation, it will either pause or play the old sample where the new one isn't yet but should be.
An xrun typically manifests itself as a (very short) discontinuity or gap in the audio signal. It might sound unpleasant, but there's nothing dangerous about it.
Examples of the latter include audio and video playback and video games. Nobody wants pauses or glitches, but if you get one once in a while, nobody dies. So people deliver these on non-RT operating systems for cost reasons.
Do you really stand by the statement of this rhetorical question? Because if yes: this attitude is a big reason for why web apps are so unpleasant to work with compared to locally running applications. Depending on the application, even 16ms vs 32ms can make a big difference.
No. This is a common misconception. The distinction between a hard realtime system and a soft realtime system is simply whether missing a timing deadline leads to a) failure of the system or b) degradation of the system (but the system continues to operate). Safety is not part of it.
Interacting with the real physical world often imposes “hard realtime” constraints (think signal processing). Whether this has safety implications simply depends on the application.
On the one hand they aren't safety critical.
On the other, I can imagine someone getting chewed out or even fired for a pause or a glitch in a professional performance.
Probably the same with live commercial video compositing.
But there are plenty of performers who apparently rely on Linux boxes and gumption.
1) There is always a possibility that something fails to run by its due date. Planes crash sometimes. Cars won't start some times. Factory machinery makes scrap parts sometimes. In a great many applications, missing a real time deadline results in degraded quality, not end of life, or regional catastrophy. The care that must be taken to lower the probability of failure needs to be in proportion to the consequence of the failure. Airplanes have redundant systems to reduce (but not eliminate) possibility of failure, while cars and trucks generally don't.
2) Even in properly working real time systems, there is a tolerance window on execution time. As machines change modes of operation, the amount of calculation effort to complete a cycle changes. If the machine is in a warm up phase, it may be doing minimal calculations, and the scan cycle is fast. Later it may be doing a quality control function that needs to do calculations on inputs from numerous sensors, and the scan cycle slows down. So long as the scan cycle doesn't exceed the limit for the process, the variation doesn't cause problems.
Much easier to just use an RTOS and save the expensive testing.
Therefore it'd be remarkable not to see a certified RTOS in such industries and applications where that validation is required, like aerospace or medical.
Most software cares about the average case, or, in the case of the Windows 10/11 start menu animation, the average across all supported machines apparently going 20 years into the future.
FWIW by-device/by-transistor-count, the bulk of "hard realtime systems" with millisecond-scale latency requirements are just audio.
The sexy stuff are all real applications too. But mostly we need this just so we don't hear pops and echos in our video calls.
[1] Actually even that's wrong: for sure there are teams of people within MS (and Apple, and anyone else in this space) measuring latency behavior at the top-level app layer and doing tuning all the way through the stack. App latency excursions can impact streams too, though ideally you have some layer of insulation there.
Even hard-realtime systems have a failure rate, in practice if not in theory - even a formally verified system might encounter a hardware bug. So it's always a case of tradeoffs between failure rate and other factors (like cost). If commodity operating systems can push their failure rate down a few orders of magnitude, that moves the needle, at least for some applications.
But it all depends on what your application is. There are a lot of applications that are "almost real-time" in need. For those, this might be useful.
But you can do all these proposed "soft" use cases with embedded Linux today. It's not like low-latency software video or audio playback is not possible, or wasn't possible twenty years ago. You only run into problems on busy systems where non-preemptible I/O could regularly get in the way. That's seldom a concern in embedded environments.
I think there are compelling reasons for making the kernel fully-preemptible, giving people more control over scheduling, and so forth. But these reasons have relatively little to do with wanting Linux to supersede minimalistic realtime OSes or bare-metal code. It's just good hygiene that will result in an OS that, even in non-RT applications, behaves better under load.
Big complex hardware is a no-no for true real-time.
That's why AbsInt and WCET tools mainly has simple CPU architectures. 8051 will truly live forever.
btw, Zephyr RTOS.
https://www.etas.com/en/applications/etas-middleware-solutio...
The bounds are usually in CPU cycles, so a faster CPU can sometimes be used even if it takes more cycles. CPUs capable of running Linux usually have higher latency (in cycles) than microcontrollers, but as long as that can be kept under the (wall clock) duration limits with bounded-time it's fine. There will still be cases where the worst-case latency to fetch from DRAM in an RT-Linux system will be higher than a slower MCU fetching from internal SRAM, so RT-Linux won't take over all these systems.
1. Suppliers have not given you sufficient information for you to be able to prove an upper bound on the time taken. (That must happen a lot.)
2. The system is so complicated that you are not totally confident of the correctness of your proof of the upper bound.
3. The only upper bound that can prove with reasonable confidence is so amazingly bad that you'd be better off with cheaper, simpler hardware.
4. There really isn't a worst case. There might, for example, be a situation equivalent to "roll the dice until you don't get snake eyes". In networking, for example, sometimes after a collision both parties try again after a random delay so the situation is resolved eventually with probability one but there's no actual upper bound. A complex CPU and memory system might have something like that? Perhaps you'd be happy with "the probability of this operation taking more than 2000 clock cycles is less than 10^-13" but perhaps not.
And already on many bigger MCUs, GPIOs are just separate blocks on a shared internal bus like AHB/APB that connects together all the chip IP, causing unpredictable latencies.
SpaceX uses x86 processors for their rockets. That small drone copter NASA put on Mars uses "big-ish" ARM cores that can probably run older versions of Android.
There are advanced real time cores like the Arm Coretex-R82. In fact many real time systems are becoming quite powerful due to the need to process and aggregate ever increasing amounts of sensor data.
68000 is the true king of realtime.
"We have Best-Effort and Guaranteed-Delivery APIs"
"I want Guaranteed Delivery!!!"
"If the GD logging interface is offline or slow, you'll take downtime; is that okay?"
"NO NO Must not take downtime!"
"If you need it logged, and can't log it, what do you do?"
These days I just point to the CAP theorem and suggest that logging is the same as any other distributed system. Because there's a wikipedia article with a triangle and the word "theorem" people seem to accept that.
[edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API]
What people probably mean is "logging is important, let's avoid losing log messages if possible", which is what "best" in "best effort" stands for. For example it's often a good idea to have a local log queue, to avoid data loss in case of a temporary log server downtime.
People often don't think precisely about what they say or want, and also often don't think through corner cases such as "what if XYZ breaks or gets slow?"
And don't get me started on "log" messages that are 300mb events. Per log. Sigh.
Having a local queue doesn't mean logging to the service is delayed, it can be sent immediately. All the local queue does is give you some resiliency, by being able to retry if the first logging attempt fails.
If an OOM happens, all bets are off anyway, since it has as much likelihood of taking out your application as it does your buffering code. The local buffering code might very well be part of the application in the first place, so the fate of the buffering code is the same as the application anyway.
It seems you're trying very hard to contrive a situation where doing nothing is better than taking reasonable steps to counter occasional network hiccups.
I think you've completely misunderstood me then. I haven't taken a stance at all on what should be done. I'm only trying to agree with the grandparent poster about logging ultimately reflecting CAP Theorem.
For companies that sell API access logs in one form or another are how bills are reconciled and usage metered.
To me, those different requirements imply that they should be treated differently by the code, probably even under distinct flows: synchronously, and ideally to somewhere that I can later compress like hell and store in some very cheap long term storage.
Whereas the debug logs that I use for debugging? Rotate out after 30 to 90d, … and yeah, best effort is fine.
(The audit logs might also end up in one's normal logs too, for convenience.)
Some systems the logs are journaled records for the business or are discoverable artifacts for compliance. In highly secure environments logs are not only durable but measures are taken to fingerprint them and their ordering (like ratchet hashing) to ensure integrity is invariant.
I would note that using disk based logging is generally harmful in these situations IMO. Network based logging is less likely to cause blocking at some OS level or other sorts of jitter that’s harder to mask. Typically I develop logging as an in memory thing that offloads to a remote service over the network. The durability of the memory store can be an issue in highly sensitive workloads, and you’ll want to do synchronous disk IO for that case to ensure durability and consistent time budgets, but for almost all application disk less logging is preferable.
Also, the failure modes of systems are very tied to durable storage devices attached to the system and very rarely to network devices. By reducing the number of things that need a disk (ideally to zero) you can remove disks from the system and its availability story. Once you get to fully disk less systems the system failure modes are actually almost nothing. But even with disks attached reducing the times you interact with the disk (especially for chatty things like logs!) reduces the likelihood the entire system fails due to a disk issue.
Depends on your failure model. I'd consider e.g. "received in memory by at least 3/5 remote servers in separate datacenters" to be safer than "committed to local disk".
In a network partition, you are either offline or your data is not consistent.
If you're writing local to your system, you're losing data if there's a single device failure.
So -- a single system collecting a log event -- it is not reliable (guaranteed) if written just to some device on that system. Instances can be de-provisioned (and logs lost), filesystems or databases can be scrambled, badguys can encrypt your data, etc.
In this context, a "network partition" prevents consistency (data not written to reliable media) or prevents availability (won't accept new requests until their activity can be logged reliably).
If you define "reliably" differently, you may have a different interpretation of log consistency.
Availability is achieved if at least one writer acknowledges a write. In a partition, it means when you have multiple parts of the system disagreeing about the write contents due to a partition in the network. But because logs are immutable and write only, this doesn’t happen in any situation. The only situation this might occur is if you’re maintaining a distributed ratchet with in delivery order semantics rather than eventually consistent temporal semantics- in which case you will never have CAP. But that’s an insanely rare edge case.
Note CAP doesn’t ensure perfect durability. I feel like you’re confusing consistency with durability. Consistency means after I’ve durably written something all nodes agree on read it’s been written. Since logs don’t support read on the online data plane this is trivially not an issue. Any write acknowledgment is sufficient.
No it doesn't. Read your own wiki link.
> In this context, a "network partition" prevents consistency (data not written to reliable media) or prevents availability (won't accept new requests until their activity can be logged reliably).
A network partition doesn't matter for a log system because there is no way to have consistency issues with logs. Even a single partitioned-off instance can accept writes without causing any problem.
Of course if you cannot connect to any instance of your log service then you cannot write logs. But that's got nothing to do with the CAP theorem.
Promtail[1] can do this if you're using Loki for logging.
[1] https://grafana.com/docs/loki/latest/send-data/promtail/
a) If your service crashes and it's in-memory, you lose logs
b) If your service can't push logs off (upstream service is down or slow) you either drop logs, run out of memory, or block
In other words it's good enough.
> [edit: added "GD" to clarify that I was referring to the guaranteed delivery logging api, not the best effort logging API]
i read GD as god-damned :-)
You can have both guaranteed delivery and no downtime if your whole system is so deterministic that anything that normally would result in blocking just will not, cannot happen. In other words it should be a hard real-time system that is formally verified top to bottom, down to the last transistor. Does anyone actually do that? Verify the program and the hardware to prove that it will never run out of memory for logs and such?
Continuing this thought, logs are probably generated endlessly, so either whoever wants them has to also guarantee that that they are processedand disposed of right after being logged... or there is a finite ammount of log messages that can be stored (arbitrary number like 10 000) but the user (of logs) has to guarantee that they will take the "mail" out of the box sooner than it overfills (at some predictable, deterministic rate). So really that means even if OUR system is mathematically perfect, we're just making the downtime someone elses problem - namely, the consumer of the infinite logs.
That, or we guarantee that the final resources of our self-contained, verified system will last longer than the finite shelf life of the system as a whole (like maybe 5 years for another arbitrary number)
From a software point of view, this system is unlikely to exist as it doesn't matter that the cause of your downtime is "something else that isn't our system". As a result, you're gonna end up requiring infinite reliable storage to upkeep your promises.
E.g., a service I wrote wrote logs to an ELK setup; we logged over TCP. But the logging was async: we didn't wait for logs to make it to ELK, and if the logging services went down, we just queued up logs locally. (To a point; at some point, the buffer fills up, and logs were discarded. The process would make a note of this if it happened, locally.)
I agree, when stuff is properly written. I don't remember the exact details, but at least with UDP the asyncness is built-in: there is no backpressure whatsoever. So poorly written software can just send udp to heart's end.
Reminds me a litte of the oldtimers trick of adding a sleep(1000) somewhere so they could later come back and have some resources later, or if they needed a quick win with the client.
Now cloud companies are using malloc(300000000) it to fake resource usage. /s
I haven’t really thought about it since then, but I can imagine lots of used cases for something like an embedded application with raspberry pi where you don’t quite want to make the leap into a microcontroller running an RTOS.
I don't recall if this was just an idea or was actually implemented. I also have seen only the one mention of RpiOS on an RTOS so I'm curious about that.
I've worked with systems that ran Linux as a task of uITRON as well as threadX, both on somewhat obscure ARM hardware. Linux managed the MMU but had a large carveout for the RTOS code. They had some strange interrupt management so that Linux could 'disable interrupts' but while Linux IRQs were disabled, an RTOS IRQ could still fire and context switch back to an RTOS task. I haven't seen anything like this on RPi though, but it's totally doable.
The sort of thing it could help with is servicing hardware that does run hard realtime. For example, you have an RTOS doing direct control of a robot or medical device or whatever, and you have a UI pendant or the like that a user is interacting with. If linux on that pendant can make some realtime latency guarantees, you may be able to simplify communication between the two without risking dropping bits on the floor.
Conversely, for the common user it could improve things like audio/video streaming, in theory but I haven't looked into details or how much trouble there is currently.
RT Linux might not pass on some certifications, but there’s likely many systems where it would be sufficient.
linux-rt's PREEMPT_RT has a negligible impact. It is there, but it is negligible. It does, however, enable a lot of use cases where Linux fails otherwise, such as pro audio.
In modern usage, it even helps reduce input jitter with videogames and enable lower latency videoconference.
I am hopeful most distributions will turn it on by default, as it benefits most users, and causes negligible impact on throughput-centric workloads.
I think I'm right in asserting that a fully-upstreamed RT kernel won't actually do anything different from a normal one unless you're actually running RT processes on it. The reason it's taken so long to upstream has been the trade-offs that have been needed to enable RT, and (per the article) there aren't many of those left.
On a BeagleBone Black, it typically gives jitter on the order of hundreds of nanoseconds. I would consider it "hard" real-time (as do they). I'm able to schedule tasks periodically on the scale of tens of microseconds, and they never get missed.
It differs from this in that Real-Time Linux attempts to make Linux itself preemptive, whereas Xenomai is essentially its own kernel, running Linux as a task on top. It provides an ABI which allows you to run your own tasks alongside or at higher prio than Linux. This sidesteps the `printk()` issue, for instance, since Xenomai doesn't care. It will gladly context switch out of printk in order to run your tasks.
The downside is that you can't make normal syscalls while inside of the Xenomai context. Well... you can, but obviously this invalidates the realtime model. For example, calling `printf()` or `malloc()` inside of a xenomai task is not preemptable. The Xenomai ABI does its best to replicate everything you may need as far as syscalls, which works great as long as you're happy doing your own heap allocations.
[1]: https://xenomai.org/
I hope it will be ready soon. I'm working in a project that has strict serial communication requirements and it has caused us a lot of headaches.
Is it other interrupts on the CPU that break your process out of the "real time" requirement, I find this all so interesting.
Is there any known upper bound on, say, how long a memory access instruction takes on x86?
But for things that really matter, I've tested by configuring the MMU to disable caching for the memory that the realtime code lives in and uses to emulate 0% hitrate. And there's usually still a fair amount of variance on top of that depending on if the memory controller has a small cache, and where the memory controller is in its refresh cycle.
For soft real time, I don't see a problem. But for hard real time, it seems a bit scary.
On some caches the line must be filled before allowing a write(ignoring any write buffer at the interface above the cache) - those basically halve the memory bandwidth when writing to a lot of cache lines. Some systems now have instructions for filling a cache line directly to avoid this. And some CPUs have bit-per-byte validity tracking to avoid this too.
Even on caches with hit-during-fill, a direct read from an address near the last-to-be-filled end of a cacheline can sometimes be a little faster than a cache miss, since the miss will fill the rest of the line first.
Intels used to boot with their caches disabled, but I haven't worked with them in forever, and never multicore.
I worked with a lot of microcontrollers, and it's not uncommon to be able to disable the instruction cache there.
There are a few things that require the data caches too, like atomic accesses on ARM. Usually we were doing something fairly short though in our realtime code, so it was easy enough to map just the memory it needed as uncacheable.
I've had to compile&install real-time kernel as a requirement for a robot arm (franka) control computer. It would be nice to know a bit more than just how to install the kernel.
Generally, having experience with Greenhills in a previous job, for personal projects like robotics or control systems I would recommend programming a microcontroller directly rather than dealing with SoC with RTOS. Modern STM32s with Cortex chips have enough processing power to run pretty much anything.
Boot time is not a bottleneck for my application (however long it takes, the client will take longer…), and I’m sure there’s some more optimal libc to use, but I’m not sure the juice is worth the squeeze.
I’m also interested in what others are doing.
Of course, this varies immensely between different use cases, as ”embedded Linux” spans such a huge swath of different kinds of systems from very cheap and simple to complex and powerful.
Unfortunately, within a kernel that’s as big as Linux, that would leave you with many, many, many counters. All of which need to be exported and monitored somehow.
That is, I would expect moving to counters to be less information, period. That not the case?
In Zephyr, we have a synchronous printk() too, as for low-level debugging and platform bringup that's usually desirable (i.e. I'd like to see the dump from just before the panic please!).
For production logging use, though, there is a fancier log system[1] designed around latency boundaries that essentially logs a minimally processed stream to a buffer than then gets flushed from a low priority thread. And this works, and avoids the kinds of problems detailed in the linked article. But it's fiddly to configure, expensive in an RTOS environment (you need RAM for that thread stack and the buffer), depends on having a I/O backend that is itself async/low-latency, and has the mentioned misfeature where when things blow up, it's usually failed to flush the information you need out of its buffer.
[1] Somewhat but not completely orthogonal with printk. Both can be implemented in terms of each others, mostly. Sometimes.
I know there was some level of argument about whether it's best to overwrite older content (ring-buffer-style, probably keeps the most important stuff) or drop messages at input time (faster, probably fewer messages dropped overall). But logging isn't my area of expertise and I forget the details.
But again, the general point being that this is a complicated problem with tradeoffs, where most developers up the stack tend to think of it as a fixed facility that shouldn't ever fail or require developer bandwidth. And it's not, it's hard.
A common logging strategy is to use a ring buffer with a separate isolated process reading from the ring. The vast majority of the time the ring buffer handles temporary disruptions (eg slow disk I/O to write messages to disk) but in the rare failure scenarios you simply overwrite events in the buffer and increment an atomic overwritten event counter. Events do not get silently dropped but you prioritize forward progress at the cost of data loss in rare scenarios.
Microkernels and pushing everything to userspace just moves the tradeoffs around. If your driver is in userspace and blocks writing a log message because the log daemon is blocked or the I/O device it is writing the log to is overloaded it does the same thing. Your realtime thread won't get what it needs from the driver within your time limit.
It all comes down to CAP theorem stuff. If you always want the kernel (or any other software) to be able to make forward progress within specific time limits then you must be willing to tolerate some data loss in failure scenarios. How much and how often it happens depends on specific design factors, memory usage, etc.
But who using a RTOS now would take the systems-integration cost/risk of switching? Would this put Android closer to Metal performance?