Intel energizes decades-old real-time Linux kernel project | Hacker News Reader

Intel energizes decades-old real-time Linux kernel project

theregister.com

136 pointsrbanffy48 comments

There are a lot of misunderstandings and shallow information in the article (and in the comments).

I used the RT_PREEMPT patches in 2012/13, so my insight is most certainly rusty, dated and might even be wrong here or there. But still:

First of all, RT_PREEMPT is about realtime as in "hard realtime" for control systems. This has nothing to do with "small latency" (the cycle time could be minutes), but with deadline guarantees, allowing control loops over (realtime) network protocols, etc.

Then, almost all important additions of the RT_PREEMPT patches were mainlined one after the other. The last bits were never mainlined although essential for full preemptive kernel AFAIR, but:

Most "modern" (>15 years) OTS-platform (x86, x86-64) is increasingly unfit for realtime - no matter what kernel/OS you use, because your CPU is not in control. Beginning with DMA, later also "management engines", modern platforms might dethrone your CPU on the lowest ring entirely and for as long as they want, to serve a request/interrupt, e.g. fan control, while your CPU wakes up 500ms later realizing that it missed quite a few deadlines. (Fan control is a bad example, but DMA is not. Also Intel ME!) This makes almost all Linux PCs inherently unable to reliably keep these guarantees.

This puts the whole "hard realtime on Linux"-project in a pickle - since it requires full control over the platform. And most shops that go this route, will take the small additional step of using something well tested and with vendor support.

And the rest buys platform and OS with support from a trusted vendor.

This hits on the core of why we still use what we do in military aerospace. COTS stuff just can’t hit the requirements nowadays.

Personally the newest X86 I’ve seen in this type of stuff was an 80286; everything newer has been POWER or SPARC of one flavor or another, typically running VXWORKS. I haven’t by any means seen everything though.

Actually, RedHawk Linux is widely used in the US “defense” industry. It supports modern hardware and even provides real-time performance when using CUDA. It is not exactly the PREEMPT_RT though.

https://concurrent-rt.com/products/software/redhawk-linux/

I have actually run into that on one or two things, but there were only peripheral to what I deal with. Drones, aircraft, etc. have all been what I mentioned in my (limited) experience.

Agreed here. I do this type work for a career, and if you need safety critical + real time (which really just means known and bounded time) guarantees, anything with an ME type device that can usurp the cores is generally a no go. Especially if the source for the firmware for said ME is not accessible.

I agree that Linux with RT_PREEMPT cannot be hard real-time in the sense that it cannot formally guarantee meeting a deadline due to CPU firmware interrupts.

However in many applications (e.g. robotics) where missing a single deadline will not result in loss of life it works perfectly fine.

The unmaskable interrupts on x86 can be a problem but long terms benchmarking can help with possible edge cases.

Other Linux-based solutions such as Xenomai & RTAI which provide even better performance deserve a mention.

There's a general misunderstanding that RT_PREEMPT somehow lowers latency. This is not at all true as RT_PREEMPT is about assured determinism not latency. Non-RT Linux has demonstrably lower latency but not determinism: response times can be very spiky (sometimes fast - often faster than RT, sometimes very slow).

I'm a big Linux fan so having realtime support in Linux would be great. But in the absence of that, Zephyr RTOS is another great open source option with some Linux kernel inspiration.

[1] https://www.zephyrproject.org/

Zephyr has nothing to do with the Linux Kernel -- especially from a technical standpoint. It is a project that is under the umbrella of the Linux Foundation, that's all.

Sorry I didn't mean to suggest that its design took inspiration from the kernel. but adjacent features like Kconfig it does.

Much of the core kernel elements definitely follow Linux kernel conventions as well. For example the `eventfd` api was copied from Linux and is quite handy. The POSIX compatibility layer constants match those in Linux, etc. The style guide is also borrowed from Linux.

And then there is stuff in Zephyr like Devicetree, which is designed to look familiar to Linux users but operates in a completely different and broken manner.

Ah, Zephyr. Another project that Intel had and then let go.

Intel is on my list, alongside Google, under the category of "companies that do one great thing, everything else is a disposable experiment that is probably already cancelled".

Proceed at your own peril with the Linutronix stuff.

What do you mean "let go"? Zephyr is supported by both Intel and linaro/arm.

I wonder about the BOM cost of Intel vs arm though, in € per "performance"? Or what else is the root cause of Intel not getting a foothold in bedded?

Zephyr started out as a WindRiver acquisition. Then Intel bought WindRiver. Then Intel sold it off.

TLDR: Intel had a foothold in embedded/realtime operating systems and let it go.

Wanna talk about XScale? Intel had an ARM license and then decided, out of the blue, to sell the product line off.

TLDR: Intel had a foothold in low-power ARM platforms and let it go.

I never went near an Edison board so I don't even address that one. And now they're doing RISC-V. Don't go near it.

So, your question is "what is the root cause of Intel not getting a foothold in (em)bedded"? The answer is Intel itself.

Yes, I know they contribute to LF and whatnot. The point is that staking your embedded development pipeline on Intel itself is something I would never do (again). And I worked with Zephyr post-Intel, it wasn't a great system. I hope it's improved since then.

All very reasonable decisions if you're an MBA, I'm sure.

Clayton Christensen plays a melancholic theme song on his angels harp to "the innovators dilemma"

Perfectly summarizes the sibling commenta on ever ongoing divestments from embedded opportunities.

Was XScale derived from StrongARM which was acquired from DEC? Because given the examples it sounds like "trust Intel to wind down acquisitions".

Of course, that still means one would be wary of betting on them in areas that aren't already well established…

It was indeed.

Intel had the top performing ARM platform with StrongARM/Xscale. They threw it away. They had their legacy embedded platforms that were popular. They threw it away. Edison could have been a compelling gateway drug for embedded Atom products. They crippled it from the start and threw it away. The root cause is MBAs making bad decisions for short term gains.

Intel did not throw it away. They sold the business unit to Marvell for $600 million.

I was working for a different CPU company at the time but the thought was by doing this it would allow Intel to focus on the sales and marketing of the Atom and push x86 everywhere rather than have a mixed x86 / ARM marketing message.

https://en.wikipedia.org/wiki/XScale

On June 27, 2006, the sale of Intel's XScale PXA mobile processor assets was announced. Intel agreed to sell the XScale PXA business to Marvell Technology Group for an estimated $600 million in cash and the assumption of unspecified liabilities. The move was intended to permit Intel to focus its resources on its core x86 and server businesses.

The move was intended to permit Intel to focus its resources on its core x86 and server businesses.

And that's where Intel continually loses out on markets because the x86 cash firehose is too good.

The XScale sale gained them $600MM, but what did they lose by being shut out of the mobile processor market for the next decade onward?

There are 2 billion iPhones on the planet somewhere and not a single one has an Intel application processor.

It doesn't strike me as abandoned. So if they no longer act as a curator while its community thrives, so what? Intel contributes to LF btw, that's some real support too.

Intel had an opportunity to develop Zephyr as a usable RTOS, internally* and externally. If they thought that the technology was compelling, why did they let it go?

*(Intel's Management Engine is still around, running....Minix. https://www.youtube.com/watch?v=iffTJ1vPCSo)

Companies are in it to make money. If it is not going anywhere, they drop it. You and I do that all the time. I have dozens of dead projects.

Cynically.. "everything else is a move designed to foul a market and disrupt a potential competitor that is then cancelled as soon as it's no longer needed."

Sight, Owner of several Intel Edison projects here, what a waste!

It’s also good to note that it’s really the only RTOS with a _real_ open source community and that has support from multiple hardware vendors. FreeRTOS is mostly whatever Amazon decides to publish. Mbed is kinda open but mostly driven by ARM. Zephyr has a community governance based on Linux’s.

What was the name of that anesthesiologist who about decade ago did a lot of work in desktop UI latency in Linux kernel? How did it went? Was any of it integrated to mainline kernel?

That's Con Kolivas. It wasn't merged, he maintained patches out of tree until recently[0]. A pity as latency doesn't seem like a high priority for Linux. Strictly speaking a real-time OS isn't about that though, that's about bounded latency, not low latency.

[0] http://ck-hack.blogspot.com/2021/08/514-and-future-of-muqss-...

I wonder if this sort of thing would add polish to the Steam Deck

(beyond basic game compatibily etc)

I would prefer romanian but you do you

https://en.m.wikipedia.org/wiki/Con_Kolivas

Nothing was merged, and last time I tried the nvidia driver didn't work with his scheduler. It's a shame, because BFS was the snappiest desktop experience I've had on Linux.

anesthesiologist

I remember quite a few small utilities and sites I've seen over the years were written by anesthesiologists, not career software developers. I wonder why.

Because the work of an anesthetist is one of the most technical ones a physician can do.

Indeed the comment I’ve had surgeons point out: often the most skilled/trained person in the operating theatre is the anesthesiologist.

can you elaborate, is it to track drug concentration profiles or work out some kind of body weight to dosage formulae or something? What's going on exactly that pushes them into the technical space.

IT is a very different field from medicine, and everyone needs a hobby.

Nuttx is an RTOS that sees a lot of use for drones and other control systems. https://nuttx.apache.org/ Posix compliant and similar feel to Linux.

It is also used as microkernel for Meadow boards running .NET.

PREEMPT_RT badly needs mainlining.

Is RT Linux a good (or at least a good enough) option for realtime OS's compared to others who where designed as RT kernels from the start?

You really have to define 'good enough'. E.g. Can handle 1000 Hz interrupt rate, and the time from firing of the interrupt timer to the time your first instruction executes is xxx microseconds. (this is important if you are doing e.g. velocity estimates from position information -- you need accurate 'dt'; therefore this time should be minimized. There are other ways to deal with jitter, but minimizing this is best.)

I turned BSD 4.3 unix into RT on a VAX 780 by using an external KW-11P clock on highest priority interrupt, have the RT code run in the kernel, and otherwise, timesharing was still going while we were running the robot -- although it was pretty sloooow. There was a big memory pool in kernel space that stored robot variables, so when the robot fell over, we would press a button which would 'freeze' the circular buffer, and then the user-space code would do an ioctl() to receive a copy of that memory pool (it was too hard to share the same pool between kernel / user space -- or maybe I was too lazy, and the solution found was 'good enough' ;-) )

You can see this in action here: https://youtu.be/mG_ZKXo6Rlg?t=34 yes, I'm sitting 'driving' the bot.

There are options. None with even close the mindshare that linux has.

>None with even close the mindshare that linux has.

When you have a hammer (Linux) in your hand, every problem looks like a nail.

seL4 can do hard realtime, with formal proofs of worst case.

Linux can't, and won't ever be able to, due to its complexity (Millions of LoCs).

RedHat has a realtime kernel.

it is available from the mirrors. Currently also available from CentOS.

last time i checked, it would not load the NVIDIA driver. YMMV.

if you are in the < 120hz range, then it will probably work fine for most non safety critical times.

>Is RT Linux a good (or at least a good enough) option for realtime OS's

Soft realtime, yes. It does keep scheduling latency under control, enabling e.g. pro audio with low latency (very small buffer). This is unlike without PREEMPT_RT, where with buffers under 20ms XRUNs happen frequently.

Hard realtime, no. It can't guarantee a thing, as the kernel is complicated (has millions of lines of code) and thus unpredictable.

Look at seL4 instead if your deadlines are hard. It has been designed for realtime, and comes with formal proofs of worst case execution time.

I was under the impression that PREEMPT_RT in being merged into mainline piece by piece already. If you look at the patches, 4.4-rt patch is 987864 bytes in size while 5.15-rt patch is only 369182 bytes. So, crudely estimating 2/3rds of -rt has already been merged to mainline between 4.4 and 5.15.

https://wiki.linuxfoundation.org/realtime/rtl/blog

Reply on news.ycombinator.com